Jul 28 01:25:18.073224 ip-10-0-140-37 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 01:25:18.484413 ip-10-0-140-37 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:18.484413 ip-10-0-140-37 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 01:25:18.484413 ip-10-0-140-37 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:18.484413 ip-10-0-140-37 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 01:25:18.484413 ip-10-0-140-37 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:18.487029 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.486850    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 01:25:18.492838 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492823    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:18.492838 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492837    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492842    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492846    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492850    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492853    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492869    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492872    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492875    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492878    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492881    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492883    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492886    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492888    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492891    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492894    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492896    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492899    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492903    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492906    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:18.492911 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492909    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492912    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492915    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492918    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492921    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492924    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492927    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492930    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492933    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492936    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492938    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492941    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492943    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492946    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492948    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492962    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492964    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492967    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492970    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492973    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:18.493385 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492975    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492977    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492980    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492983    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492986    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492989    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492991    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492993    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492996    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.492999    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493001    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493003    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493006    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493008    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493012    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493014    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493017    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493019    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493023    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493026    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:18.494065 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493029    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493032    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493035    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493038    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493040    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493043    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493046    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493049    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493052    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493054    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493057    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493059    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493062    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493065    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493067    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493069    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493072    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493074    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493077    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:18.494781 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493079    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493081    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493084    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493087    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493089    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493091    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493094    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493097    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493099    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493103    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493106    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493109    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493112    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493114    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493116    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493119    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493121    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493124    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493126    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:18.495624 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.493129    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495197    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495212    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495218    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495222    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495226    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495231    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495236    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495240    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495244    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495250    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495254    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495258    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495263    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495267    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495272    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495275    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495280    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495284    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495288    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:18.496425 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495292    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495297    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495302    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495307    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495312    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495320    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495327    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495332    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495337    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495342    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495347    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495352    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495359    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495366    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495371    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495376    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495380    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495384    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495389    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:18.497371 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495393    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495397    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495401    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495405    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495410    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495414    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495419    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495423    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495427    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495431    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495436    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495440    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495445    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495450    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495455    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495459    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495463    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495468    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495472    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495476    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:18.498219 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495480    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495484    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495488    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495492    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495497    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495501    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495505    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495510    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495514    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495519    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495523    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495527    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495532    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495536    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495540    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495544    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495548    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495554    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495558    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495562    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:18.498805 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495567    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495571    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495575    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495579    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495584    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495590    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495594    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495599    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495603    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495607    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495611    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495615    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495619    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495623    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495626    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495631    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495634    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495638    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495642    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:18.499396 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495649    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.495654    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496267    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496277    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496283    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496289    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496294    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496300    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496304    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496309    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496314    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496319    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496323    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496328    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496332    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496337    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496341    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496346    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496351    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:18.500040 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496356    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496360    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496364    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496368    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496372    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496377    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496381    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496385    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496389    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496393    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496397    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496401    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496406    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496412    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496416    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496421    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496424    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496429    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496434    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:18.500573 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496438    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496442    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496446    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496451    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496455    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496459    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496463    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496467    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496472    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496477    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496481    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496485    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496489    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496494    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496498    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496503    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496507    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496511    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496515    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496520    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:18.501076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496525    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496529    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496533    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496537    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496542    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496546    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496554    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496560    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496564    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496569    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496574    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496579    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496583    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496588    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496592    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496597    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496601    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496605    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496609    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496613    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:18.501647 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496617    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496622    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496626    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496632    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496636    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496640    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496647    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496651    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496655    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496660    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496664    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496668    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496672    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496676    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496680    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496685    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496689    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496693    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496699    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496703    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:18.502222 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496708    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496712    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.496716    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496822    2556 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496833    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496842    2556 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496849    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496855    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496869    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496876    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496882    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496887    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496892    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496897    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496902    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496907    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496912    2556 flags.go:64] FLAG: --cgroup-root=""
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496917    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496921    2556 flags.go:64] FLAG: --client-ca-file=""
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496930    2556 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496935    2556 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496943    2556 flags.go:64] FLAG: --cluster-domain=""
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496947    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 01:25:18.502815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496974    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496980    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496985    2556 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496991    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.496996    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497000    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497006    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497013    2556 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497018    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497023    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497027    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497032    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497038    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497049    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497055    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497060    2556 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497065    2556 flags.go:64] FLAG: --enable-server="true"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497070    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497076    2556 flags.go:64] FLAG: --event-burst="100"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497081    2556 flags.go:64] FLAG: --event-qps="50"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497085    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497091    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497095    2556 flags.go:64] FLAG: --eviction-hard=""
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497101    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 01:25:18.503386 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497106    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497111    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497116    2556 flags.go:64] FLAG: --eviction-soft=""
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497120    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497124    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497130    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497136    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497140    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497145    2556 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497150    2556 flags.go:64] FLAG: --feature-gates=""
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497156    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497160    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497165    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497170    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497175    2556 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497182    2556 flags.go:64] FLAG: --help="false"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497187    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497192    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497197    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497202    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497207    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497215    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497220    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 01:25:18.503977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497224    2556 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497230    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497234    2556 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497243    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497248    2556 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497253    2556 flags.go:64] FLAG: --kube-reserved=""
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497258    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497263    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497268    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497272    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497277    2556 flags.go:64] FLAG: --lock-file=""
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497282    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497286    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497291    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497300    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497305    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497309    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497314    2556 flags.go:64] FLAG: --logging-format="text"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497319    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497324    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497328    2556 flags.go:64] FLAG: --manifest-url=""
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497332    2556 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497345    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497350    2556 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497358    2556 flags.go:64] FLAG: --max-pods="110"
Jul 28 01:25:18.504632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497362    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497367    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497371    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497376    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497381    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497386    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497390    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497401    2556 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497406    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497411    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497416    2556 flags.go:64] FLAG: --pod-cidr=""
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497421    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497429    2556 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497433    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497438    2556 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497442    2556 flags.go:64] FLAG: --port="10250"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497447    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497452    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03db4f405975d8d3a"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497458    2556 flags.go:64] FLAG: --qos-reserved=""
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497463    2556 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497467    2556 flags.go:64] FLAG: --register-node="true"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497472    2556 flags.go:64] FLAG: --register-with-taints=""
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497477    2556 flags.go:64] FLAG: --registry-burst="10"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497483    2556 flags.go:64] FLAG: --registry-qps="5"
Jul 28 01:25:18.505276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497487    2556 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497492    2556 flags.go:64] FLAG: --reserved-memory=""
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497498    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497503    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497507    2556 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497512    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497516    2556 flags.go:64] FLAG: --runonce="false"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497521    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497526    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497531    2556 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497535    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497540    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497545    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497551    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497555    2556 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497560    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497565    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497569    2556 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497573    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497578    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497583    2556 flags.go:64] FLAG: --system-cgroups=""
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497588    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497596    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497600    2556 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497605    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 01:25:18.505856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497612    2556 flags.go:64] FLAG: --tls-min-version=""
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497616    2556 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497621    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497625    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497630    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497634    2556 flags.go:64] FLAG: --v="2"
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497640    2556 flags.go:64] FLAG: --version="false"
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497647    2556 flags.go:64] FLAG: --vmodule=""
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497653    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.497659    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497865    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497873    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497878    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497883    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497888    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497893    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497898    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497902    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497907    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497912    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497917    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497922    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:18.506474 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497926    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497930    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497934    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497939    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497943    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497947    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497973    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497978    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497982    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497986    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497990    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497994    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.497998    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498005    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498010    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498016    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498020    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498024    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498030    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:18.507032 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498034    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498039    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498043    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498047    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498051    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498055    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498061    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498065    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498069    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498073    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498077    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498084    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498088    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498092    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498096    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498100    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498104    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498108    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498113    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498117    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:18.507497 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498121    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498125    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498129    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498133    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498137    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498141    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498145    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498149    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498153    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498157    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498161    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498166    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498170    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498174    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498178    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498183    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498187    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498191    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498195    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498199    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:18.508088 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498203    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498207    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498211    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498218    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498222    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498226    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498229    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498233    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498237    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498240    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498245    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498251    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498256    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498261    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498265    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498269    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498273    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498277    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498281    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498286    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:18.508605 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498291    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498295    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498299    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498304    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498308    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498313    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498319    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.498325    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.499053    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.505984    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.505999    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.506019    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.506025    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506079    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:18.509115 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506083    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506086    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506089    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506092    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506095    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506098    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506101    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506105    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506108    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506110    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506113    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506115    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506118    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506121    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506124    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506126    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506128    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506131    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506133    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506136    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:18.509468 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506138    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506141    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506143    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506145    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506148    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506151    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506154    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506156    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506160    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506163    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506165    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506168    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506170    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506173    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506175    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506178    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506181    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506184    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506186    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506190    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:18.509938 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506194    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506197    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506200    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506202    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506205    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506208    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506210    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506213    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506216    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506218    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506221    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506223    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506226    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506229    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506233    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506237    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506240    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506243    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506246    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:18.510440 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506249    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506253    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506257    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506260    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506263    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506266    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506269    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506271    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506274    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506277    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506280    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506282    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506285    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506288    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506290    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506293    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506295    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506298    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506300    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506303    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:18.510891 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506305    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506308    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506311    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506313    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506315    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506318    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506321    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506323    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506326    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506328    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506331    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506334    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506337    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506340    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506343    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506346    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506348    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506351    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:18.511441 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506353    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.506358    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506467    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506471    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506474    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506477    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506480    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506483    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506486    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506489    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506492    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506494    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506497    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506499    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506502    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:18.511899 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506505    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506508    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506511    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506514    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506517    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506519    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506522    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506524    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506527    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506529    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506532    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506534    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506537    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506540    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506542    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506545    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506547    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506550    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506552    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506555    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:18.512289 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506557    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506560    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506562    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506565    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506567    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506569    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506572    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506574    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506576    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506579    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506581    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506584    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506586    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506588    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506591    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506593    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506596    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506598    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506600    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506603    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:18.512776 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506607    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506610    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506612    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506615    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506618    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506621    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506625    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506629    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506632    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506634    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506637    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506640    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506642    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506645    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506647    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506649    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506652    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506654    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506657    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:18.513294 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506659    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506661    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506664    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506666    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506669    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506671    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506674    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506676    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506679    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506681    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506684    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506686    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506688    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506691    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506693    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506696    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506698    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506701    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506703    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506705    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:18.513736 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506708    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506710    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506713    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506716    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506718    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506720    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:18.506723    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.506728    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.506834    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.509064    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.510259    2556 server.go:1019] "Starting client certificate rotation"
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.510390    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.510424    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:18.514217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.512798    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:18.515171 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.515157    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:18.529003 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.528980    2556 log.go:25] "Validated CRI v1 runtime API"
Jul 28 01:25:18.531018 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.530993    2556 log.go:25] "Validated CRI v1 image API"
Jul 28 01:25:18.531826 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.531808    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 01:25:18.535914 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.535896    2556 fs.go:135] Filesystem UUIDs: map[4858c310-bdcd-45ea-9dca-b757ade924de:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a938fad3-9cda-46ed-8ce3-2db6a3b15506:/dev/nvme0n1p3]
Jul 28 01:25:18.536000 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.535914    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 01:25:18.540282 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.540262    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:18.541674 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.541565    2556 manager.go:216] Machine: {Timestamp:2026-07-28 01:25:18.539557648 +0000 UTC m=+0.344907128 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3080177 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b94597b1cf4668a2738331de62d61 SystemUUID:ec2b9459-7b1c-f466-8a27-38331de62d61 BootID:0372eee5-49b5-4059-93b4-a5652cfd747a Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5f:71:db:ec:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5f:71:db:ec:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:47:7c:24:27:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 01:25:18.541674 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.541665    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 01:25:18.541797 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.541741    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 01:25:18.545226 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.545199    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 01:25:18.545369 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.545228    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 01:25:18.546834 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.546824    2556 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 01:25:18.546871 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.546836    2556 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 01:25:18.546871 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.546843    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:18.547573 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.547562    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:18.548468 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.548453    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 01:25:18.550770 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.550755    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:18.550908 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.550897    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 01:25:18.550995 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.550937    2556 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 01:25:18.550995 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.550974    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 01:25:18.551093 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.551001    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 01:25:18.551093 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.551023    2556 kubelet.go:398] "Adding apiserver pod source"
Jul 28 01:25:18.551093 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.551039    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 01:25:18.551485 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.551470    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:18.551559 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.551494    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:18.555339 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.555321    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 01:25:18.557164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.557149    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 01:25:18.557205 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.557181    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 01:25:18.558534 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558521    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 01:25:18.558570 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558543    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 01:25:18.558570 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558553    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 01:25:18.558570 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558562    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 01:25:18.558670 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558576    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 01:25:18.558670 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558589    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 01:25:18.558670 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558602    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 01:25:18.558670 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558611    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 01:25:18.558670 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558631    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 01:25:18.558670 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558641    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 01:25:18.558670 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558653    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 01:25:18.558670 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.558667    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 01:25:18.559518 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.559506    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 01:25:18.559558 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.559520    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 01:25:18.562932 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.562915    2556 server.go:1268] "Started kubelet"
Jul 28 01:25:18.563068 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.563011    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 01:25:18.563383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.562996    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 01:25:18.563467 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.563434    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 01:25:18.563817 ip-10-0-140-37 systemd[1]: Started Kubernetes Kubelet.
Jul 28 01:25:18.565138 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.565119    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 01:25:18.569261 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.569235    2556 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 01:25:18.569632 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.569574    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 01:25:18.571171 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.571145    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 01:25:18.571171 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.571151    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 01:25:18.571304 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.571207    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 01:25:18.571935 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.571097    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-37.ec2.internal.18c64fafc049032a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-37.ec2.internal,UID:ip-10-0-140-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-37.ec2.internal,},FirstTimestamp:2026-07-28 01:25:18.562861866 +0000 UTC m=+0.368211346,LastTimestamp:2026-07-28 01:25:18.562861866 +0000 UTC m=+0.368211346,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-37.ec2.internal,}"
Jul 28 01:25:18.574518 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.574499    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 01:25:18.574595 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.574512    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:18.574656 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.574623    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:18.575158 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575139    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 01:25:18.575304 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575284    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 01:25:18.575383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575336    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 01:25:18.575452 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575408    2556 factory.go:55] Registering systemd factory
Jul 28 01:25:18.575452 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575449    2556 factory.go:223] Registration of the systemd container factory successfully
Jul 28 01:25:18.575544 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575461    2556 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 01:25:18.575544 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575471    2556 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 01:25:18.575697 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575680    2556 factory.go:153] Registering CRI-O factory
Jul 28 01:25:18.575778 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575707    2556 factory.go:223] Registration of the crio container factory successfully
Jul 28 01:25:18.575831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575782    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 01:25:18.575831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575807    2556 factory.go:103] Registering Raw factory
Jul 28 01:25:18.575831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.575822    2556 manager.go:1196] Started watching for new ooms in manager
Jul 28 01:25:18.576226 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.576213    2556 manager.go:349] Starting recovery of all containers
Jul 28 01:25:18.583763 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.583722    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 01:25:18.585161 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.585131    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 01:25:18.585244 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.585184    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 01:25:18.588592 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.588573    2556 manager.go:354] Recovery completed
Jul 28 01:25:18.592587 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.592574    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:18.594620 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.594562    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:18.594620 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.594602    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:18.594620 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.594616    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:18.596135 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.596118    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 01:25:18.596135 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.596130    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:18.596135 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.596133    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 01:25:18.596274 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.596151    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:18.596854 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.596841    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:18.596966 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.596873    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:18.596966 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.596883    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:18.597088 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.597070    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lt6vz"
Jul 28 01:25:18.598299 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.598288    2556 policy_none.go:49] "None policy: Start"
Jul 28 01:25:18.598337 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.598304    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 01:25:18.598337 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.598315    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 01:25:18.599633 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.599620    2556 policy_none.go:47] "Start"
Jul 28 01:25:18.604427 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.604408    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lt6vz"
Jul 28 01:25:18.605941 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.605879    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-37.ec2.internal.18c64fafc22d13eb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-37.ec2.internal,UID:ip-10-0-140-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-37.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-37.ec2.internal,},FirstTimestamp:2026-07-28 01:25:18.594585579 +0000 UTC m=+0.399935063,LastTimestamp:2026-07-28 01:25:18.594585579 +0000 UTC m=+0.399935063,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-37.ec2.internal,}"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.643382    2556 manager.go:337] "Starting Device Plugin manager"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.643584    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.643599    2556 server.go:87] "Starting device plugin registration server"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.643859    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.643871    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.643989    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.644074    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.644083    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.644602    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.644662    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.647710    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.647747    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.647767    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.647800    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 01:25:18.653514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.649749    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:18.744807 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.744725    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:18.745679 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.745664    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:18.745735 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.745694    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:18.745735 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.745711    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:18.745816 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.745739    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.748777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.748761    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal"]
Jul 28 01:25:18.748830 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.748821    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:18.749553 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.749520    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:18.749553 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.749556    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:18.749683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.749567    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:18.750945 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.750933    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:18.751054 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.751042    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.751099 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.751069    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:18.751625 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.751610    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:18.751625 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.751619    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:18.751754 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.751636    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:18.751754 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.751643    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:18.751754 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.751650    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:18.751754 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.751654    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:18.753043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.753019    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.753043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.753044    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:18.753651 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.753634    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:18.753741 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.753662    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:18.753741 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.753676    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:18.755049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.755034    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.755105 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.755054    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-37.ec2.internal\": node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:18.776488 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.776465    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/281101458b4f0d4fa8d3f138f29e0e85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal\" (UID: \"281101458b4f0d4fa8d3f138f29e0e85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.776565 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.776492    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/281101458b4f0d4fa8d3f138f29e0e85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal\" (UID: \"281101458b4f0d4fa8d3f138f29e0e85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.776565 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.776507    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae809c78f68e15731047d4f8c293c8a9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-37.ec2.internal\" (UID: \"ae809c78f68e15731047d4f8c293c8a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.778436 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.778421    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:18.784464 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.784449    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-37.ec2.internal\" not found" node="ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.788685 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.788670    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-37.ec2.internal\" not found" node="ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.877273 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.877252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/281101458b4f0d4fa8d3f138f29e0e85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal\" (UID: \"281101458b4f0d4fa8d3f138f29e0e85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.877383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.877315    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/281101458b4f0d4fa8d3f138f29e0e85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal\" (UID: \"281101458b4f0d4fa8d3f138f29e0e85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.877383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.877276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/281101458b4f0d4fa8d3f138f29e0e85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal\" (UID: \"281101458b4f0d4fa8d3f138f29e0e85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.877383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.877355    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/281101458b4f0d4fa8d3f138f29e0e85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal\" (UID: \"281101458b4f0d4fa8d3f138f29e0e85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.877383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.877372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae809c78f68e15731047d4f8c293c8a9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-37.ec2.internal\" (UID: \"ae809c78f68e15731047d4f8c293c8a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.877519 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:18.877399    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae809c78f68e15731047d4f8c293c8a9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-37.ec2.internal\" (UID: \"ae809c78f68e15731047d4f8c293c8a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:18.879371 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.879353    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:18.979423 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:18.979392    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:19.080218 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.080154    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:19.087329 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.087312    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:19.092591 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.092577    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:19.180851 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.180825    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:19.281354 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.281329    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:19.381812 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.381742    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-37.ec2.internal\" not found"
Jul 28 01:25:19.395375 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.395356    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:19.475280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.475249    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:19.487864 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.487843    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:19.489730 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.489716    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal"
Jul 28 01:25:19.497060 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.497042    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:19.509763 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.509744    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 01:25:19.509879 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.509857    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:19.552015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.551996    2556 apiserver.go:52] "Watching apiserver"
Jul 28 01:25:19.561238 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.561220    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 01:25:19.561723 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.561699    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths","openshift-cluster-node-tuning-operator/tuned-kmchd","openshift-dns/node-resolver-pjclb","openshift-image-registry/node-ca-st5g5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal","openshift-multus/multus-additional-cni-plugins-4cdnz","openshift-multus/multus-hgt5v","kube-system/global-pull-secret-syncer-hwr6q","kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal","openshift-multus/network-metrics-daemon-9qd9t","openshift-network-diagnostics/network-check-target-d8c8n","openshift-network-operator/iptables-alerter-6ktzj","openshift-ovn-kubernetes/ovnkube-node-wrg24","kube-system/konnectivity-agent-bftn8"]
Jul 28 01:25:19.563415 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.563396    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.564764 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.564748    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:19.564838 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.564817    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:19.566218 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.566203    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.566524 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.566380    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 01:25:19.566524 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.566419    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 01:25:19.566753 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.566576    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 01:25:19.566753 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.566589    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 01:25:19.566753 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.566595    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wqv29\""
Jul 28 01:25:19.567445 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.567432    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.570058 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.570027    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 01:25:19.570164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.570089    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 01:25:19.570447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.570299    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-58d66\""
Jul 28 01:25:19.570447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.570325    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 01:25:19.570447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.570385    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 01:25:19.570447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.570420    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:19.570674 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.570513    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pbjmw\""
Jul 28 01:25:19.570973 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.570830    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 01:25:19.571656 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.571639    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.573008 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.572987    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pgqqx\""
Jul 28 01:25:19.573099 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.573033    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 01:25:19.573099 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.573063    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:19.573205 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.573106    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 01:25:19.573256 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.573217    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:19.573811 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.573794    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 01:25:19.573865 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.573833    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 01:25:19.573906 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.573867    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-prmw9\""
Jul 28 01:25:19.574696 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.574647    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:19.574696 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.574664    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.576033 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.576016    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.576723 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.576706    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hqvhw\""
Jul 28 01:25:19.576817 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.576792    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 01:25:19.577014 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.576993    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 01:25:19.577225 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.577206    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 01:25:19.577300 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.577272    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 01:25:19.577510 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.577493    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 01:25:19.577582 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.577522    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 01:25:19.577582 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.577496    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:19.577677 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.577612    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:19.578552 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.578535    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:19.578622 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.578602    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:19.579176 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.579163    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4xr5b\""
Jul 28 01:25:19.580041 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.580023    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.581270 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581251    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-log-socket\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.581365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581285    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-sysctl-d\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.581365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581309    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-cni-dir\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.581365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581350    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-socket-dir-parent\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.581365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581360    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581395    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-run-netns\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581424    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-var-lib-cni-bin\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581452    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxtw2\" (UniqueName: \"kubernetes.io/projected/56abc673-f9ba-4b33-a82f-d0469ba37eb5-kube-api-access-bxtw2\") pod \"node-ca-st5g5\" (UID: \"56abc673-f9ba-4b33-a82f-d0469ba37eb5\") " pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581476    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-run-systemd\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69a3339f-758b-4280-98bc-1bde23005e7b-env-overrides\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581525    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-sysconfig\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581548    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-lib-modules\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581570    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-run-openvswitch\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581641    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2305f190-18a0-46c6-a291-264be71b6d0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.581692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581690    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69a3339f-758b-4280-98bc-1bde23005e7b-ovn-node-metrics-cert\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69a3339f-758b-4280-98bc-1bde23005e7b-ovnkube-script-lib\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581730    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-modprobe-d\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581746    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-sys\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581776    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c6d42fc-71f1-466d-9183-521c2b330a67-cni-binary-copy\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581800    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56abc673-f9ba-4b33-a82f-d0469ba37eb5-serviceca\") pod \"node-ca-st5g5\" (UID: \"56abc673-f9ba-4b33-a82f-d0469ba37eb5\") " pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581818    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-os-release\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581847    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69a3339f-758b-4280-98bc-1bde23005e7b-ovnkube-config\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581900    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-run-k8s-cni-cncf-io\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.581977    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrv2\" (UniqueName: \"kubernetes.io/projected/8c6d42fc-71f1-466d-9183-521c2b330a67-kube-api-access-rqrv2\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582008    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2305f190-18a0-46c6-a291-264be71b6d0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582032    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-etc-openvswitch\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-sysctl-conf\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582078    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-tuned\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582104    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5ba0f8-327b-4257-97da-632040885d84-tmp-dir\") pod \"node-resolver-pjclb\" (UID: \"cc5ba0f8-327b-4257-97da-632040885d84\") " pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582126    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56abc673-f9ba-4b33-a82f-d0469ba37eb5-host\") pod \"node-ca-st5g5\" (UID: \"56abc673-f9ba-4b33-a82f-d0469ba37eb5\") " pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.582182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582166    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-systemd\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582199    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-system-cni-dir\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-node-log\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582244    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-etc-kubernetes\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d20181f-51d1-42ae-b14e-411cda629c02-agent-certs\") pod \"konnectivity-agent-bftn8\" (UID: \"2d20181f-51d1-42ae-b14e-411cda629c02\") " pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582286    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-cni-bin\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582307    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-var-lib-cni-multus\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6674r\" (UniqueName: \"kubernetes.io/projected/2305f190-18a0-46c6-a291-264be71b6d0d-kube-api-access-6674r\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582349    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582406    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-run-multus-certs\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582428    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-os-release\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582460    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-hostroot\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582478    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-daemon-config\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582523    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-slash\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.582683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582548    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncpf\" (UniqueName: \"kubernetes.io/projected/69a3339f-758b-4280-98bc-1bde23005e7b-kube-api-access-vncpf\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a4c582-5124-454c-93ec-e7b7c6377d29-tmp\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582604    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-conf-dir\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582635    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pp7g\" (UniqueName: \"kubernetes.io/projected/cc5ba0f8-327b-4257-97da-632040885d84-kube-api-access-2pp7g\") pod \"node-resolver-pjclb\" (UID: \"cc5ba0f8-327b-4257-97da-632040885d84\") " pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582656    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92m2\" (UniqueName: \"kubernetes.io/projected/e9a4c582-5124-454c-93ec-e7b7c6377d29-kube-api-access-w92m2\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582672    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-cnibin\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582696    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582721    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-systemd-units\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-var-lib-openvswitch\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582758    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-run-ovn\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582774    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-cni-netd\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582796    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-kubernetes\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582817    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-system-cni-dir\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582864    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-run\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582893    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-var-lib-kubelet\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-host\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582948    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-var-lib-kubelet\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.583164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.582986    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc5ba0f8-327b-4257-97da-632040885d84-hosts-file\") pod \"node-resolver-pjclb\" (UID: \"cc5ba0f8-327b-4257-97da-632040885d84\") " pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583008    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d20181f-51d1-42ae-b14e-411cda629c02-konnectivity-ca\") pod \"konnectivity-agent-bftn8\" (UID: \"2d20181f-51d1-42ae-b14e-411cda629c02\") " pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583032    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-cnibin\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2305f190-18a0-46c6-a291-264be71b6d0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-kubelet\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583084    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-run-netns\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583100    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583133    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-kubelet-config\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583154    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-dbus\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583273    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583330    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583370    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583702    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 01:25:19.583893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.583840    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kpl6s\""
Jul 28 01:25:19.584269 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.584061    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:19.584304 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.584281    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k7tk6\""
Jul 28 01:25:19.584349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.584336    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:19.587810 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.587793    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:19.609408 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.609371    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 01:20:20 +0000 UTC" deadline="2028-04-15 13:20:15.940115732 +0000 UTC"
Jul 28 01:25:19.609408 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.609407    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15059h54m56.330710729s"
Jul 28 01:25:19.618290 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.618271    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxgg8"
Jul 28 01:25:19.626758 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.626741    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxgg8"
Jul 28 01:25:19.660976 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.660930    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281101458b4f0d4fa8d3f138f29e0e85.slice/crio-19af96bba30af88fcc6754cc893910210bec47ac70f87e39ba7c4a5000d50fba WatchSource:0}: Error finding container 19af96bba30af88fcc6754cc893910210bec47ac70f87e39ba7c4a5000d50fba: Status 404 returned error can't find the container with id 19af96bba30af88fcc6754cc893910210bec47ac70f87e39ba7c4a5000d50fba
Jul 28 01:25:19.661151 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.661138    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae809c78f68e15731047d4f8c293c8a9.slice/crio-0289edc907daa79af08d658b80b65ff8b68f7de4f22519fff779898ea9b78cea WatchSource:0}: Error finding container 0289edc907daa79af08d658b80b65ff8b68f7de4f22519fff779898ea9b78cea: Status 404 returned error can't find the container with id 0289edc907daa79af08d658b80b65ff8b68f7de4f22519fff779898ea9b78cea
Jul 28 01:25:19.667155 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.667070    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:25:19.675793 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.675774    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 01:25:19.683390 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683369    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-cni-dir\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.683475 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683394    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-socket-dir-parent\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.683475 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683409    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-run-netns\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.683475 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683424    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-var-lib-cni-bin\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.683475 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683464    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-var-lib-cni-bin\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683484    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxtw2\" (UniqueName: \"kubernetes.io/projected/56abc673-f9ba-4b33-a82f-d0469ba37eb5-kube-api-access-bxtw2\") pod \"node-ca-st5g5\" (UID: \"56abc673-f9ba-4b33-a82f-d0469ba37eb5\") " pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683487    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-run-netns\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-run-systemd\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683532    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-socket-dir-parent\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683534    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-run-systemd\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683544    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-cni-dir\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69a3339f-758b-4280-98bc-1bde23005e7b-env-overrides\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683576    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-sysconfig\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683592    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-lib-modules\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683618    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zbs\" (UniqueName: \"kubernetes.io/projected/294ecef6-7cbe-446d-96f6-36de1ab923aa-kube-api-access-h2zbs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-run-openvswitch\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683653    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2305f190-18a0-46c6-a291-264be71b6d0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.683683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683662    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-sysconfig\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683678    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69a3339f-758b-4280-98bc-1bde23005e7b-ovn-node-metrics-cert\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683701    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-run-openvswitch\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683730    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69a3339f-758b-4280-98bc-1bde23005e7b-ovnkube-script-lib\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-lib-modules\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683753    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-modprobe-d\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-sys\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683801    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c6d42fc-71f1-466d-9183-521c2b330a67-cni-binary-copy\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56abc673-f9ba-4b33-a82f-d0469ba37eb5-serviceca\") pod \"node-ca-st5g5\" (UID: \"56abc673-f9ba-4b33-a82f-d0469ba37eb5\") " pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683841    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-os-release\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683864    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69a3339f-758b-4280-98bc-1bde23005e7b-ovnkube-config\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683865    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-modprobe-d\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683901    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-run-k8s-cni-cncf-io\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683929    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrv2\" (UniqueName: \"kubernetes.io/projected/8c6d42fc-71f1-466d-9183-521c2b330a67-kube-api-access-rqrv2\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.683981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-run-k8s-cni-cncf-io\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684004    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2305f190-18a0-46c6-a291-264be71b6d0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684031    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-etc-openvswitch\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684053    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684157    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-sysctl-conf\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684215    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69a3339f-758b-4280-98bc-1bde23005e7b-env-overrides\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684245    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-etc-openvswitch\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684272    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2305f190-18a0-46c6-a291-264be71b6d0d-cni-binary-copy\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-sys\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684350    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69a3339f-758b-4280-98bc-1bde23005e7b-ovnkube-script-lib\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684377    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-os-release\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684057    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-sysctl-conf\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684443    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69a3339f-758b-4280-98bc-1bde23005e7b-ovnkube-config\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684468    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-tuned\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684496    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5ba0f8-327b-4257-97da-632040885d84-tmp-dir\") pod \"node-resolver-pjclb\" (UID: \"cc5ba0f8-327b-4257-97da-632040885d84\") " pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56abc673-f9ba-4b33-a82f-d0469ba37eb5-host\") pod \"node-ca-st5g5\" (UID: \"56abc673-f9ba-4b33-a82f-d0469ba37eb5\") " pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684547    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7759363-2ec3-41df-99fc-64fea8326c20-host-slash\") pod \"iptables-alerter-6ktzj\" (UID: \"a7759363-2ec3-41df-99fc-64fea8326c20\") " pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-systemd\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684592    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-system-cni-dir\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684615    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-node-log\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684657    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-etc-kubernetes\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.684985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684662    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2305f190-18a0-46c6-a291-264be71b6d0d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684671    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56abc673-f9ba-4b33-a82f-d0469ba37eb5-serviceca\") pod \"node-ca-st5g5\" (UID: \"56abc673-f9ba-4b33-a82f-d0469ba37eb5\") " pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d20181f-51d1-42ae-b14e-411cda629c02-agent-certs\") pod \"konnectivity-agent-bftn8\" (UID: \"2d20181f-51d1-42ae-b14e-411cda629c02\") " pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684708    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-cni-bin\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684726    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56abc673-f9ba-4b33-a82f-d0469ba37eb5-host\") pod \"node-ca-st5g5\" (UID: \"56abc673-f9ba-4b33-a82f-d0469ba37eb5\") " pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684734    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-var-lib-cni-multus\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6674r\" (UniqueName: \"kubernetes.io/projected/2305f190-18a0-46c6-a291-264be71b6d0d-kube-api-access-6674r\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684769    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c6d42fc-71f1-466d-9183-521c2b330a67-cni-binary-copy\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684783    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-etc-kubernetes\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684788    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684820    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-node-log\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684826    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-var-lib-cni-multus\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684835    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-run-multus-certs\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684870    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-systemd\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684869    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a7759363-2ec3-41df-99fc-64fea8326c20-iptables-alerter-script\") pod \"iptables-alerter-6ktzj\" (UID: \"a7759363-2ec3-41df-99fc-64fea8326c20\") " pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-system-cni-dir\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684911    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-os-release\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.685985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684976    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-hostroot\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-daemon-config\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685022    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-run-multus-certs\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685031    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-device-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.684882    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-cni-bin\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-etc-selinux\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685088    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k68sb\" (UniqueName: \"kubernetes.io/projected/a7759363-2ec3-41df-99fc-64fea8326c20-kube-api-access-k68sb\") pod \"iptables-alerter-6ktzj\" (UID: \"a7759363-2ec3-41df-99fc-64fea8326c20\") " pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685115    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685143    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-slash\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685156    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5ba0f8-327b-4257-97da-632040885d84-tmp-dir\") pod \"node-resolver-pjclb\" (UID: \"cc5ba0f8-327b-4257-97da-632040885d84\") " pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685172    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncpf\" (UniqueName: \"kubernetes.io/projected/69a3339f-758b-4280-98bc-1bde23005e7b-kube-api-access-vncpf\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685195    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a4c582-5124-454c-93ec-e7b7c6377d29-tmp\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685218    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-conf-dir\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685245    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pp7g\" (UniqueName: \"kubernetes.io/projected/cc5ba0f8-327b-4257-97da-632040885d84-kube-api-access-2pp7g\") pod \"node-resolver-pjclb\" (UID: \"cc5ba0f8-327b-4257-97da-632040885d84\") " pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685269    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-socket-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-registration-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685329    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbk4\" (UniqueName: \"kubernetes.io/projected/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-kube-api-access-vdbk4\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.686777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685353    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92m2\" (UniqueName: \"kubernetes.io/projected/e9a4c582-5124-454c-93ec-e7b7c6377d29-kube-api-access-w92m2\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-cnibin\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685401    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-systemd-units\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685449    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-var-lib-openvswitch\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685459    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-os-release\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685470    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-run-ovn\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-cni-netd\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685514    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-run-ovn\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685532    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-kubernetes\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-system-cni-dir\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685563    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-conf-dir\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685582    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-run\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-var-lib-kubelet\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685630    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-host\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685666    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-var-lib-kubelet\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685699    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc5ba0f8-327b-4257-97da-632040885d84-hosts-file\") pod \"node-resolver-pjclb\" (UID: \"cc5ba0f8-327b-4257-97da-632040885d84\") " pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685730    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d20181f-51d1-42ae-b14e-411cda629c02-konnectivity-ca\") pod \"konnectivity-agent-bftn8\" (UID: \"2d20181f-51d1-42ae-b14e-411cda629c02\") " pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:19.687556 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685765    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-cnibin\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685806    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2305f190-18a0-46c6-a291-264be71b6d0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685852    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.685874    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-host-var-lib-kubelet\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.685972    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.686031    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret podName:ffc9bda4-ee09-499b-bb30-db6c295d9ec2 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:20.18601052 +0000 UTC m=+1.991360009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret") pod "global-pull-secret-syncer-hwr6q" (UID: "ffc9bda4-ee09-499b-bb30-db6c295d9ec2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-slash\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686388    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c6d42fc-71f1-466d-9183-521c2b330a67-multus-daemon-config\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686400    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-cnibin\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686444    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-system-cni-dir\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686442    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d20181f-51d1-42ae-b14e-411cda629c02-konnectivity-ca\") pod \"konnectivity-agent-bftn8\" (UID: \"2d20181f-51d1-42ae-b14e-411cda629c02\") " pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686483    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-systemd-units\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686529    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686537    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-kubernetes\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686564    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-cni-netd\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686581    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-run\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-kubelet\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686616    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-var-lib-openvswitch\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-run-netns\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.686649    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c6d42fc-71f1-466d-9183-521c2b330a67-hostroot\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687201    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2305f190-18a0-46c6-a291-264be71b6d0d-cnibin\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc5ba0f8-327b-4257-97da-632040885d84-hosts-file\") pod \"node-resolver-pjclb\" (UID: \"cc5ba0f8-327b-4257-97da-632040885d84\") " pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687321    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-var-lib-kubelet\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687332    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2305f190-18a0-46c6-a291-264be71b6d0d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687353    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-host\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687438    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-sys-fs\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687459    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-kubelet-config\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687504    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-dbus\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687536    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-log-socket\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687561    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-sysctl-d\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687562    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-tuned\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687585    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687633    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.688813 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687686    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-run-netns\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.689385 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687716    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-log-socket\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.689385 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687721    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69a3339f-758b-4280-98bc-1bde23005e7b-host-kubelet\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.689385 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687736    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-kubelet-config\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:19.689385 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687747    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69a3339f-758b-4280-98bc-1bde23005e7b-ovn-node-metrics-cert\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.689385 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687790    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-dbus\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:19.689385 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687798    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9a4c582-5124-454c-93ec-e7b7c6377d29-etc-sysctl-d\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.689385 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.687853    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d20181f-51d1-42ae-b14e-411cda629c02-agent-certs\") pod \"konnectivity-agent-bftn8\" (UID: \"2d20181f-51d1-42ae-b14e-411cda629c02\") " pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:19.689600 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.689487    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9a4c582-5124-454c-93ec-e7b7c6377d29-tmp\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.691615 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.691594    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:19.691716 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.691623    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:19.691716 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.691635    2556 projected.go:196] Error preparing data for projected volume kube-api-access-48rfx for pod openshift-network-diagnostics/network-check-target-d8c8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:19.691716 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.691689    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx podName:68629e90-6236-451d-8615-a3fac3628ea0 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:20.191672709 +0000 UTC m=+1.997022193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-48rfx" (UniqueName: "kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx") pod "network-check-target-d8c8n" (UID: "68629e90-6236-451d-8615-a3fac3628ea0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:19.692227 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.692201    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxtw2\" (UniqueName: \"kubernetes.io/projected/56abc673-f9ba-4b33-a82f-d0469ba37eb5-kube-api-access-bxtw2\") pod \"node-ca-st5g5\" (UID: \"56abc673-f9ba-4b33-a82f-d0469ba37eb5\") " pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.692569 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.692552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrv2\" (UniqueName: \"kubernetes.io/projected/8c6d42fc-71f1-466d-9183-521c2b330a67-kube-api-access-rqrv2\") pod \"multus-hgt5v\" (UID: \"8c6d42fc-71f1-466d-9183-521c2b330a67\") " pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.693147 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.693120    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6674r\" (UniqueName: \"kubernetes.io/projected/2305f190-18a0-46c6-a291-264be71b6d0d-kube-api-access-6674r\") pod \"multus-additional-cni-plugins-4cdnz\" (UID: \"2305f190-18a0-46c6-a291-264be71b6d0d\") " pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.694042 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.694017    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pp7g\" (UniqueName: \"kubernetes.io/projected/cc5ba0f8-327b-4257-97da-632040885d84-kube-api-access-2pp7g\") pod \"node-resolver-pjclb\" (UID: \"cc5ba0f8-327b-4257-97da-632040885d84\") " pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.695641 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.695622    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92m2\" (UniqueName: \"kubernetes.io/projected/e9a4c582-5124-454c-93ec-e7b7c6377d29-kube-api-access-w92m2\") pod \"tuned-kmchd\" (UID: \"e9a4c582-5124-454c-93ec-e7b7c6377d29\") " pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.696044 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.696026    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncpf\" (UniqueName: \"kubernetes.io/projected/69a3339f-758b-4280-98bc-1bde23005e7b-kube-api-access-vncpf\") pod \"ovnkube-node-wrg24\" (UID: \"69a3339f-758b-4280-98bc-1bde23005e7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.788586 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-sys-fs\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.788751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788594    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.788751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zbs\" (UniqueName: \"kubernetes.io/projected/294ecef6-7cbe-446d-96f6-36de1ab923aa-kube-api-access-h2zbs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:19.788751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788633    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-sys-fs\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.788751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788655    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:19.788751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788668    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.788751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788680    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7759363-2ec3-41df-99fc-64fea8326c20-host-slash\") pod \"iptables-alerter-6ktzj\" (UID: \"a7759363-2ec3-41df-99fc-64fea8326c20\") " pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.788751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7759363-2ec3-41df-99fc-64fea8326c20-host-slash\") pod \"iptables-alerter-6ktzj\" (UID: \"a7759363-2ec3-41df-99fc-64fea8326c20\") " pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.788786    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a7759363-2ec3-41df-99fc-64fea8326c20-iptables-alerter-script\") pod \"iptables-alerter-6ktzj\" (UID: \"a7759363-2ec3-41df-99fc-64fea8326c20\") " pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:19.788850    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs podName:294ecef6-7cbe-446d-96f6-36de1ab923aa nodeName:}" failed. No retries permitted until 2026-07-28 01:25:20.288829569 +0000 UTC m=+2.094179051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs") pod "network-metrics-daemon-9qd9t" (UID: "294ecef6-7cbe-446d-96f6-36de1ab923aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788878    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-device-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788907    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-etc-selinux\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788931    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k68sb\" (UniqueName: \"kubernetes.io/projected/a7759363-2ec3-41df-99fc-64fea8326c20-kube-api-access-k68sb\") pod \"iptables-alerter-6ktzj\" (UID: \"a7759363-2ec3-41df-99fc-64fea8326c20\") " pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788940    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-device-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788976    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-socket-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.788999    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-registration-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.789025    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbk4\" (UniqueName: \"kubernetes.io/projected/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-kube-api-access-vdbk4\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.789071    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-etc-selinux\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.789143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.789127    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-registration-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.789593 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.789194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-socket-dir\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.789593 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.789432    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a7759363-2ec3-41df-99fc-64fea8326c20-iptables-alerter-script\") pod \"iptables-alerter-6ktzj\" (UID: \"a7759363-2ec3-41df-99fc-64fea8326c20\") " pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.798677 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.798648    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k68sb\" (UniqueName: \"kubernetes.io/projected/a7759363-2ec3-41df-99fc-64fea8326c20-kube-api-access-k68sb\") pod \"iptables-alerter-6ktzj\" (UID: \"a7759363-2ec3-41df-99fc-64fea8326c20\") " pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:19.800383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.800357    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zbs\" (UniqueName: \"kubernetes.io/projected/294ecef6-7cbe-446d-96f6-36de1ab923aa-kube-api-access-h2zbs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:19.800609 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.800591    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbk4\" (UniqueName: \"kubernetes.io/projected/d2ba6f30-8e92-4a5e-9c35-9d50bd85d517-kube-api-access-vdbk4\") pod \"aws-ebs-csi-driver-node-wnths\" (UID: \"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.865645 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.865619    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:19.898149 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.898096    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hgt5v"
Jul 28 01:25:19.906435 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.906414    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6d42fc_71f1_466d_9183_521c2b330a67.slice/crio-aa2022ef308517b21679d3f23313a45dd049c263a021e0251a7427709971c632 WatchSource:0}: Error finding container aa2022ef308517b21679d3f23313a45dd049c263a021e0251a7427709971c632: Status 404 returned error can't find the container with id aa2022ef308517b21679d3f23313a45dd049c263a021e0251a7427709971c632
Jul 28 01:25:19.913464 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.913445    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pjclb"
Jul 28 01:25:19.921010 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.920990    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5ba0f8_327b_4257_97da_632040885d84.slice/crio-05dde885f4c1e1e963073800f328467b41ea9fc9bd1838c024b007a6d66c76da WatchSource:0}: Error finding container 05dde885f4c1e1e963073800f328467b41ea9fc9bd1838c024b007a6d66c76da: Status 404 returned error can't find the container with id 05dde885f4c1e1e963073800f328467b41ea9fc9bd1838c024b007a6d66c76da
Jul 28 01:25:19.930373 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.930354    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-st5g5"
Jul 28 01:25:19.937862 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.937841    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56abc673_f9ba_4b33_a82f_d0469ba37eb5.slice/crio-c44fb2009988eaa410bc62333837797acf002c37e099ba09b1ea457277c50be3 WatchSource:0}: Error finding container c44fb2009988eaa410bc62333837797acf002c37e099ba09b1ea457277c50be3: Status 404 returned error can't find the container with id c44fb2009988eaa410bc62333837797acf002c37e099ba09b1ea457277c50be3
Jul 28 01:25:19.942616 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.942603    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:19.949744 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.949722    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d20181f_51d1_42ae_b14e_411cda629c02.slice/crio-eef56e2743cd08255c0645bafed709b29144af537b68dc7a924468b564d1545b WatchSource:0}: Error finding container eef56e2743cd08255c0645bafed709b29144af537b68dc7a924468b564d1545b: Status 404 returned error can't find the container with id eef56e2743cd08255c0645bafed709b29144af537b68dc7a924468b564d1545b
Jul 28 01:25:19.968328 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.968312    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4cdnz"
Jul 28 01:25:19.974315 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.974296    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:19.975295 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.975273    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2305f190_18a0_46c6_a291_264be71b6d0d.slice/crio-3d3ea3da489baa864a354b59b38ca42d259637135dff03ed66d14220135ffa6d WatchSource:0}: Error finding container 3d3ea3da489baa864a354b59b38ca42d259637135dff03ed66d14220135ffa6d: Status 404 returned error can't find the container with id 3d3ea3da489baa864a354b59b38ca42d259637135dff03ed66d14220135ffa6d
Jul 28 01:25:19.980272 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.980253    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kmchd"
Jul 28 01:25:19.981220 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.981201    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a3339f_758b_4280_98bc_1bde23005e7b.slice/crio-ea67d9228468abff8895ac3de17cbc1f3bd57e66778dfce7fc2ecc7b9bdf94b1 WatchSource:0}: Error finding container ea67d9228468abff8895ac3de17cbc1f3bd57e66778dfce7fc2ecc7b9bdf94b1: Status 404 returned error can't find the container with id ea67d9228468abff8895ac3de17cbc1f3bd57e66778dfce7fc2ecc7b9bdf94b1
Jul 28 01:25:19.987077 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.987057    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a4c582_5124_454c_93ec_e7b7c6377d29.slice/crio-008bcbd1961d94c2eac9734f801b85c4e34079e94092ddad2d382799c2dc87d0 WatchSource:0}: Error finding container 008bcbd1961d94c2eac9734f801b85c4e34079e94092ddad2d382799c2dc87d0: Status 404 returned error can't find the container with id 008bcbd1961d94c2eac9734f801b85c4e34079e94092ddad2d382799c2dc87d0
Jul 28 01:25:19.987790 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:19.987772    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths"
Jul 28 01:25:19.996762 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:19.995894    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ba6f30_8e92_4a5e_9c35_9d50bd85d517.slice/crio-5078f93c39a7a3387d5c8e4c64739e65b0195c3a0af8a02e03746f3df7b8ec70 WatchSource:0}: Error finding container 5078f93c39a7a3387d5c8e4c64739e65b0195c3a0af8a02e03746f3df7b8ec70: Status 404 returned error can't find the container with id 5078f93c39a7a3387d5c8e4c64739e65b0195c3a0af8a02e03746f3df7b8ec70
Jul 28 01:25:20.006019 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:20.006003    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6ktzj"
Jul 28 01:25:20.012876 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:20.012850    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7759363_2ec3_41df_99fc_64fea8326c20.slice/crio-e00bf0acf1b6510dfdae614b5a8f8413c8ba068b5c4d779da22328735685b28f WatchSource:0}: Error finding container e00bf0acf1b6510dfdae614b5a8f8413c8ba068b5c4d779da22328735685b28f: Status 404 returned error can't find the container with id e00bf0acf1b6510dfdae614b5a8f8413c8ba068b5c4d779da22328735685b28f
Jul 28 01:25:20.140097 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:20.140069    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:20.192911 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:20.192822    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:20.192911 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:20.192886    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:20.193080 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:20.193011    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:20.193080 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:20.193021    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:20.193080 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:20.193036    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:20.193080 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:20.193049    2556 projected.go:196] Error preparing data for projected volume kube-api-access-48rfx for pod openshift-network-diagnostics/network-check-target-d8c8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:20.193080 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:20.193076    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret podName:ffc9bda4-ee09-499b-bb30-db6c295d9ec2 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:21.193058823 +0000 UTC m=+2.998408307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret") pod "global-pull-secret-syncer-hwr6q" (UID: "ffc9bda4-ee09-499b-bb30-db6c295d9ec2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:20.193279 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:20.193094    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx podName:68629e90-6236-451d-8615-a3fac3628ea0 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:21.193084379 +0000 UTC m=+2.998433847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-48rfx" (UniqueName: "kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx") pod "network-check-target-d8c8n" (UID: "68629e90-6236-451d-8615-a3fac3628ea0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:20.293926 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:20.293888    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:20.294140 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:20.294100    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:20.294208 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:20.294169    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs podName:294ecef6-7cbe-446d-96f6-36de1ab923aa nodeName:}" failed. No retries permitted until 2026-07-28 01:25:21.29415053 +0000 UTC m=+3.099500011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs") pod "network-metrics-daemon-9qd9t" (UID: "294ecef6-7cbe-446d-96f6-36de1ab923aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:22.261815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.261751    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:21 +0000 UTC" deadline="2028-05-13 23:02:31.596902076 +0000 UTC"
Jul 28 01:25:22.261815 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.261812    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15741h37m9.335094442s"
Jul 28 01:25:22.284632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.284605    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:22.284798 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:22.284744    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:22.304468 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.304411    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hgt5v" event={"ID":"8c6d42fc-71f1-466d-9183-521c2b330a67","Type":"ContainerStarted","Data":"aa2022ef308517b21679d3f23313a45dd049c263a021e0251a7427709971c632"}
Jul 28 01:25:22.319978 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.319947    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal" event={"ID":"ae809c78f68e15731047d4f8c293c8a9","Type":"ContainerStarted","Data":"0289edc907daa79af08d658b80b65ff8b68f7de4f22519fff779898ea9b78cea"}
Jul 28 01:25:22.329123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.328828    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6ktzj" event={"ID":"a7759363-2ec3-41df-99fc-64fea8326c20","Type":"ContainerStarted","Data":"e00bf0acf1b6510dfdae614b5a8f8413c8ba068b5c4d779da22328735685b28f"}
Jul 28 01:25:22.335827 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.335767    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths" event={"ID":"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517","Type":"ContainerStarted","Data":"5078f93c39a7a3387d5c8e4c64739e65b0195c3a0af8a02e03746f3df7b8ec70"}
Jul 28 01:25:22.340531 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.340470    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" event={"ID":"2305f190-18a0-46c6-a291-264be71b6d0d","Type":"ContainerStarted","Data":"3d3ea3da489baa864a354b59b38ca42d259637135dff03ed66d14220135ffa6d"}
Jul 28 01:25:22.351745 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.351716    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bftn8" event={"ID":"2d20181f-51d1-42ae-b14e-411cda629c02","Type":"ContainerStarted","Data":"eef56e2743cd08255c0645bafed709b29144af537b68dc7a924468b564d1545b"}
Jul 28 01:25:22.365310 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.365283    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal" event={"ID":"281101458b4f0d4fa8d3f138f29e0e85","Type":"ContainerStarted","Data":"19af96bba30af88fcc6754cc893910210bec47ac70f87e39ba7c4a5000d50fba"}
Jul 28 01:25:22.389818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.389789    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kmchd" event={"ID":"e9a4c582-5124-454c-93ec-e7b7c6377d29","Type":"ContainerStarted","Data":"008bcbd1961d94c2eac9734f801b85c4e34079e94092ddad2d382799c2dc87d0"}
Jul 28 01:25:22.400710 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.400673    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerStarted","Data":"ea67d9228468abff8895ac3de17cbc1f3bd57e66778dfce7fc2ecc7b9bdf94b1"}
Jul 28 01:25:22.405590 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.405545    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-st5g5" event={"ID":"56abc673-f9ba-4b33-a82f-d0469ba37eb5","Type":"ContainerStarted","Data":"c44fb2009988eaa410bc62333837797acf002c37e099ba09b1ea457277c50be3"}
Jul 28 01:25:22.428238 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.428173    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pjclb" event={"ID":"cc5ba0f8-327b-4257-97da-632040885d84","Type":"ContainerStarted","Data":"05dde885f4c1e1e963073800f328467b41ea9fc9bd1838c024b007a6d66c76da"}
Jul 28 01:25:22.560184 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.560037    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:22.833519 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.833431    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:22.833519 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.833499    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:22.833750 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:22.833669    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:22.833750 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:22.833729    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret podName:ffc9bda4-ee09-499b-bb30-db6c295d9ec2 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:24.833711146 +0000 UTC m=+5.006456380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret") pod "global-pull-secret-syncer-hwr6q" (UID: "ffc9bda4-ee09-499b-bb30-db6c295d9ec2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:22.834157 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:22.834137    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:22.834230 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:22.834163    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:22.834230 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:22.834176    2556 projected.go:196] Error preparing data for projected volume kube-api-access-48rfx for pod openshift-network-diagnostics/network-check-target-d8c8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:22.834230 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:22.834218    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx podName:68629e90-6236-451d-8615-a3fac3628ea0 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:24.834204387 +0000 UTC m=+5.006949621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-48rfx" (UniqueName: "kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx") pod "network-check-target-d8c8n" (UID: "68629e90-6236-451d-8615-a3fac3628ea0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:22.934658 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:22.934622    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:22.934838 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:22.934747    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:22.934838 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:22.934803    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs podName:294ecef6-7cbe-446d-96f6-36de1ab923aa nodeName:}" failed. No retries permitted until 2026-07-28 01:25:24.934784812 +0000 UTC m=+5.107530045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs") pod "network-metrics-daemon-9qd9t" (UID: "294ecef6-7cbe-446d-96f6-36de1ab923aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:23.262817 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:23.262699    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:21 +0000 UTC" deadline="2028-04-10 18:11:15.278053556 +0000 UTC"
Jul 28 01:25:23.262817 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:23.262759    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14944h45m52.015298174s"
Jul 28 01:25:23.281820 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:23.281143    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:23.281820 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:23.281292    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:23.281820 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:23.281693    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:23.281820 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:23.281779    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:24.283808 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:24.283296    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:24.283808 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:24.283437    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:24.850680 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:24.850641    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:24.850879 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:24.850746    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:24.850941 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:24.850893    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:24.850941 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:24.850917    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:24.850941 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:24.850929    2556 projected.go:196] Error preparing data for projected volume kube-api-access-48rfx for pod openshift-network-diagnostics/network-check-target-d8c8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:24.851048 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:24.850984    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx podName:68629e90-6236-451d-8615-a3fac3628ea0 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:28.850963986 +0000 UTC m=+9.023709218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-48rfx" (UniqueName: "kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx") pod "network-check-target-d8c8n" (UID: "68629e90-6236-451d-8615-a3fac3628ea0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:24.851351 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:24.851330    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:24.851445 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:24.851390    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret podName:ffc9bda4-ee09-499b-bb30-db6c295d9ec2 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:28.851373478 +0000 UTC m=+9.024118707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret") pod "global-pull-secret-syncer-hwr6q" (UID: "ffc9bda4-ee09-499b-bb30-db6c295d9ec2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:24.952206 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:24.951590    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:24.952206 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:24.951775    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:24.952206 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:24.951845    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs podName:294ecef6-7cbe-446d-96f6-36de1ab923aa nodeName:}" failed. No retries permitted until 2026-07-28 01:25:28.951823836 +0000 UTC m=+9.124569082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs") pod "network-metrics-daemon-9qd9t" (UID: "294ecef6-7cbe-446d-96f6-36de1ab923aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:25.281296 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:25.280933    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:25.281296 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:25.281062    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:25.281509 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:25.281398    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:25.281509 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:25.281477    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:26.283383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:26.283350    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:26.283845 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:26.283486    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:27.281020 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:27.280981    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:27.281191 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:27.280990    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:27.281191 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:27.281121    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:27.281293 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:27.281243    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:28.281292 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:28.281256    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:28.281771 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:28.281454    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:28.887473 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:28.887425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:28.887654 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:28.887510    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:28.887728 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:28.887675    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:28.887728 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:28.887700    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:28.887728 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:28.887712    2556 projected.go:196] Error preparing data for projected volume kube-api-access-48rfx for pod openshift-network-diagnostics/network-check-target-d8c8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:28.887868 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:28.887770    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx podName:68629e90-6236-451d-8615-a3fac3628ea0 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:36.887751764 +0000 UTC m=+17.060497004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-48rfx" (UniqueName: "kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx") pod "network-check-target-d8c8n" (UID: "68629e90-6236-451d-8615-a3fac3628ea0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:28.888146 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:28.888128    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:28.888235 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:28.888177    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret podName:ffc9bda4-ee09-499b-bb30-db6c295d9ec2 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:36.888166445 +0000 UTC m=+17.060911685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret") pod "global-pull-secret-syncer-hwr6q" (UID: "ffc9bda4-ee09-499b-bb30-db6c295d9ec2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:28.988402 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:28.988365    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:28.988587 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:28.988542    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:28.988664 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:28.988624    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs podName:294ecef6-7cbe-446d-96f6-36de1ab923aa nodeName:}" failed. No retries permitted until 2026-07-28 01:25:36.988610608 +0000 UTC m=+17.161355848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs") pod "network-metrics-daemon-9qd9t" (UID: "294ecef6-7cbe-446d-96f6-36de1ab923aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:29.280832 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:29.280756    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:29.281005 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:29.280870    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:29.281090 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:29.280766    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:29.281221 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:29.281138    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:30.283792 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:30.283766    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:30.284148 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:30.283856    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:31.281605 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:31.281575    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:31.281605 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:31.281604    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:31.281816 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:31.281710    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:31.282127 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:31.282103    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:32.280671 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:32.280641    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:32.281121 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:32.280774    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:33.281103 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:33.281078    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:33.281663 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:33.281192    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:33.281663 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:33.281272    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:33.281663 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:33.281379    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:34.281131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:34.281095    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:34.281550 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:34.281225    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:35.281609 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:35.281571    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:35.281609 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:35.281603    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:35.282104 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:35.281699    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:35.282104 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:35.281814    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:36.281671 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:36.281104    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:36.281671 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:36.281248    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:36.955051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:36.955020    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:36.955241 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:36.955061    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:36.955241 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:36.955168    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:36.955241 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:36.955183    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:36.955241 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:36.955202    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:36.955241 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:36.955211    2556 projected.go:196] Error preparing data for projected volume kube-api-access-48rfx for pod openshift-network-diagnostics/network-check-target-d8c8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:36.955241 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:36.955219    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret podName:ffc9bda4-ee09-499b-bb30-db6c295d9ec2 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:52.955204001 +0000 UTC m=+33.127949232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret") pod "global-pull-secret-syncer-hwr6q" (UID: "ffc9bda4-ee09-499b-bb30-db6c295d9ec2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:36.955426 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:36.955249    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx podName:68629e90-6236-451d-8615-a3fac3628ea0 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:52.955237721 +0000 UTC m=+33.127982949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-48rfx" (UniqueName: "kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx") pod "network-check-target-d8c8n" (UID: "68629e90-6236-451d-8615-a3fac3628ea0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:37.055729 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:37.055702    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:37.055885 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:37.055870    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:37.055957 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:37.055941    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs podName:294ecef6-7cbe-446d-96f6-36de1ab923aa nodeName:}" failed. No retries permitted until 2026-07-28 01:25:53.055920683 +0000 UTC m=+33.228665925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs") pod "network-metrics-daemon-9qd9t" (UID: "294ecef6-7cbe-446d-96f6-36de1ab923aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:37.281502 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:37.281407    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:37.281675 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:37.281407    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:37.281675 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:37.281549    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:37.281675 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:37.281641    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:38.280870 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:38.280833    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:38.281040 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:38.280964    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:39.281519 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:39.281487    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:39.281981 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:39.281625    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:39.281981 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:39.281699    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:39.281981 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:39.281801    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:40.281900 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:40.281866    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:40.282331 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:40.281987    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:41.280926 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:41.280896    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:41.281076 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:41.280896    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:41.281076 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:41.281003    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:41.281076 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:41.281061    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:42.281791 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.281628    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:42.282103 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:42.281876    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:42.479025 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.478660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal" event={"ID":"ae809c78f68e15731047d4f8c293c8a9","Type":"ContainerStarted","Data":"014097d5541704b76861ed2631f521041004980c986211c21b3bfb38677ed0ed"}
Jul 28 01:25:42.480406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.480369    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kmchd" event={"ID":"e9a4c582-5124-454c-93ec-e7b7c6377d29","Type":"ContainerStarted","Data":"edc985cf5c31d16f6e9c40da8deeb1ee67dbb999c4c4b87fa9e5e955c9ba603d"}
Jul 28 01:25:42.483700 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.483683    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:25:42.484104 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.484084    2556 generic.go:356] "Generic (PLEG): container finished" podID="69a3339f-758b-4280-98bc-1bde23005e7b" containerID="7f97edb3616022c7d7186ae81aefce125f1c004628f692ac17ca1d1e85cbd69a" exitCode=1
Jul 28 01:25:42.484189 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.484160    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerStarted","Data":"57eec5cd0a03645e7b9c0a441fb6d18f0ec0ccbf40199edfab894b4be45f850c"}
Jul 28 01:25:42.484248 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.484237    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerDied","Data":"7f97edb3616022c7d7186ae81aefce125f1c004628f692ac17ca1d1e85cbd69a"}
Jul 28 01:25:42.484306 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.484290    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerStarted","Data":"6db5e7da04effe9cbd59d9096f9adba696365f325b5658dcfb936907937ce6ad"}
Jul 28 01:25:42.485806 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.485787    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hgt5v" event={"ID":"8c6d42fc-71f1-466d-9183-521c2b330a67","Type":"ContainerStarted","Data":"dc11041b8ba46fc6fc54b07e1bcd11d3153e6e5572a5f2f9073d2e3b8f1c13cc"}
Jul 28 01:25:42.496924 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.496872    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal" podStartSLOduration=21.496853919 podStartE2EDuration="21.496853919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:42.495778296 +0000 UTC m=+22.668523546" watchObservedRunningTime="2026-07-28 01:25:42.496853919 +0000 UTC m=+22.669599168"
Jul 28 01:25:42.518659 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.518586    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kmchd" podStartSLOduration=2.637680227 podStartE2EDuration="22.518546931s" totalImagesPullingTime="19.880866704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:42.517573154 +0000 UTC m=+22.690318394" watchObservedRunningTime="2026-07-28 01:25:42.518546931 +0000 UTC m=+22.691292183"
Jul 28 01:25:42.541917 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:42.540539    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hgt5v" podStartSLOduration=2.569639673 podStartE2EDuration="22.540515956s" totalImagesPullingTime="19.970876283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:42.54031902 +0000 UTC m=+22.713064269" watchObservedRunningTime="2026-07-28 01:25:42.540515956 +0000 UTC m=+22.713261205"
Jul 28 01:25:43.281123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.281038    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:43.281123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.281076    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:43.281293 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:43.281181    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:43.281335 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:43.281292    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:43.489677 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.489640    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-st5g5" event={"ID":"56abc673-f9ba-4b33-a82f-d0469ba37eb5","Type":"ContainerStarted","Data":"62f81d1877bf4fec8dafbfd9ba3212fdda7c384754da7466ea6dbf4abf466479"}
Jul 28 01:25:43.490974 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.490950    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pjclb" event={"ID":"cc5ba0f8-327b-4257-97da-632040885d84","Type":"ContainerStarted","Data":"6736bb7680991524fe19e398dd40ad65059e3ed7714fd2b64f301d767c331c2a"}
Jul 28 01:25:43.492306 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.492286    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6ktzj" event={"ID":"a7759363-2ec3-41df-99fc-64fea8326c20","Type":"ContainerStarted","Data":"7fffaadb7edcc27d60803ef72f4d18115e786d69db19df81e0fbd534ea81e96e"}
Jul 28 01:25:43.493676 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.493650    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths" event={"ID":"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517","Type":"ContainerStarted","Data":"64f358ae54ac698ec8b3866594b06661386096ed756d33319b59f09337da80f1"}
Jul 28 01:25:43.495111 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.495090    2556 generic.go:356] "Generic (PLEG): container finished" podID="2305f190-18a0-46c6-a291-264be71b6d0d" containerID="c18516078b4d24d6b1d27d658d17ef324ec8d8dea15610e7fd2cfd7d7a754a40" exitCode=0
Jul 28 01:25:43.495198 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.495155    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" event={"ID":"2305f190-18a0-46c6-a291-264be71b6d0d","Type":"ContainerDied","Data":"c18516078b4d24d6b1d27d658d17ef324ec8d8dea15610e7fd2cfd7d7a754a40"}
Jul 28 01:25:43.496523 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.496441    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bftn8" event={"ID":"2d20181f-51d1-42ae-b14e-411cda629c02","Type":"ContainerStarted","Data":"70f4c40672b10abc38e06a67ed9bf8589377e011d009b3249a04def0a8004c7c"}
Jul 28 01:25:43.497927 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.497903    2556 generic.go:356] "Generic (PLEG): container finished" podID="281101458b4f0d4fa8d3f138f29e0e85" containerID="5efc4ed8722ee4997929aa4a2ccfea1413c8d08802c7e9df68792dfbcd6f7a29" exitCode=0
Jul 28 01:25:43.498018 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.497983    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal" event={"ID":"281101458b4f0d4fa8d3f138f29e0e85","Type":"ContainerDied","Data":"5efc4ed8722ee4997929aa4a2ccfea1413c8d08802c7e9df68792dfbcd6f7a29"}
Jul 28 01:25:43.500667 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.500650    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:25:43.501001 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.500982    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerStarted","Data":"282a7e4ba4588063d98d58f34f38d875cdf2148f981f2e31e292a6e8a64631e5"}
Jul 28 01:25:43.501094 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.501044    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerStarted","Data":"8caa1ec50641d389f86e9e8b489e08242736fc87f55a552031d9fc1826b55a52"}
Jul 28 01:25:43.501094 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.501090    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerStarted","Data":"d490cffcfe2ca9f369e8a042962d7918ce072c92e793d7f8dfbf4d68d79f7a0d"}
Jul 28 01:25:43.505065 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.505029    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-st5g5" podStartSLOduration=3.578153504 podStartE2EDuration="23.505018181s" totalImagesPullingTime="19.926864677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:43.50476012 +0000 UTC m=+23.677505370" watchObservedRunningTime="2026-07-28 01:25:43.505018181 +0000 UTC m=+23.677763469"
Jul 28 01:25:43.539036 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.538364    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pjclb" podStartSLOduration=3.641222066 podStartE2EDuration="23.538348855s" totalImagesPullingTime="19.897126789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:43.51879183 +0000 UTC m=+23.691537079" watchObservedRunningTime="2026-07-28 01:25:43.538348855 +0000 UTC m=+23.711094105"
Jul 28 01:25:43.564288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.564251    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bftn8" podStartSLOduration=3.671732726 podStartE2EDuration="23.564239033s" totalImagesPullingTime="19.892506307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:43.53874275 +0000 UTC m=+23.711488000" watchObservedRunningTime="2026-07-28 01:25:43.564239033 +0000 UTC m=+23.736984281"
Jul 28 01:25:43.597546 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.597502    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6ktzj" podStartSLOduration=3.768677011 podStartE2EDuration="23.597486725s" totalImagesPullingTime="19.828809714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:43.580882903 +0000 UTC m=+23.753628162" watchObservedRunningTime="2026-07-28 01:25:43.597486725 +0000 UTC m=+23.770231967"
Jul 28 01:25:43.763197 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:43.763176    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 01:25:44.283678 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:44.283639    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:44.283952 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:44.283783    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:44.289797 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:44.289701    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T01:25:43.763194611Z","UUID":"d913552d-4177-48a0-8c2c-4cbb53f43db6","Handler":null,"Name":"","Endpoint":""}
Jul 28 01:25:44.291535 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:44.291511    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 01:25:44.291680 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:44.291573    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 01:25:44.506506 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:44.506383    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths" event={"ID":"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517","Type":"ContainerStarted","Data":"b6ee2a96c72d587aa5df287918e27755b255c2733177889f1b4119c2516a8791"}
Jul 28 01:25:44.508750 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:44.508715    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal" event={"ID":"281101458b4f0d4fa8d3f138f29e0e85","Type":"ContainerStarted","Data":"13d9a1d95febb50fe0c3feb2a0039b07044aa7b1d09fc8588ddafc12f8a0352d"}
Jul 28 01:25:45.281385 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.281351    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:45.281698 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.281400    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:45.281698 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:45.281492    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:45.281698 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:45.281639    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:45.449956 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.449915    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:45.450443 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.450419    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:45.465752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.465705    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-37.ec2.internal" podStartSLOduration=24.465686069 podStartE2EDuration="24.465686069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:44.526730229 +0000 UTC m=+24.699475480" watchObservedRunningTime="2026-07-28 01:25:45.465686069 +0000 UTC m=+25.638431317"
Jul 28 01:25:45.513187 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.513119    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths" event={"ID":"d2ba6f30-8e92-4a5e-9c35-9d50bd85d517","Type":"ContainerStarted","Data":"b247e17105469458a287e7bc57506fe2244ec18f23adbe18cdd90f4b627545ff"}
Jul 28 01:25:45.516939 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.516918    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:25:45.517370 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.517342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerStarted","Data":"d0284910de2e7afbcfeee59c8866c7a5b447a2096bd4cd98418ade8d51be3726"}
Jul 28 01:25:45.517601 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.517582    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:45.518041 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.518019    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bftn8"
Jul 28 01:25:45.530779 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:45.530735    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnths" podStartSLOduration=3.5453951569999997 podStartE2EDuration="25.530721571s" totalImagesPullingTime="21.985326414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:45.530393451 +0000 UTC m=+25.703138719" watchObservedRunningTime="2026-07-28 01:25:45.530721571 +0000 UTC m=+25.703466819"
Jul 28 01:25:46.281053 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:46.281026    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:46.281301 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:46.281167    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:47.280948 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:47.280862    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:47.281381 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:47.280993    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:47.281381 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:47.281059    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:47.281381 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:47.281159    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:48.281305 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:48.281111    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:48.282021 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:48.281405    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:48.526671 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:48.526643    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:25:48.527038 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:48.527006    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerStarted","Data":"448eb89fa5c391ca4cfe15c1d99a3f8e8fca1807a8bd67b22f61baf9370823d1"}
Jul 28 01:25:48.527261 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:48.527236    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:48.527464 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:48.527439    2556 scope.go:117] "RemoveContainer" containerID="7f97edb3616022c7d7186ae81aefce125f1c004628f692ac17ca1d1e85cbd69a"
Jul 28 01:25:48.528898 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:48.528872    2556 generic.go:356] "Generic (PLEG): container finished" podID="2305f190-18a0-46c6-a291-264be71b6d0d" containerID="19541ceb282fb82277fbad67f48ede822f677922d863ba99056b64a2025a9d24" exitCode=0
Jul 28 01:25:48.528966 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:48.528917    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" event={"ID":"2305f190-18a0-46c6-a291-264be71b6d0d","Type":"ContainerDied","Data":"19541ceb282fb82277fbad67f48ede822f677922d863ba99056b64a2025a9d24"}
Jul 28 01:25:48.544111 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:48.544092    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:49.281001 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.280934    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:49.281152 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:49.281038    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:49.281152 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.281090    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:49.281229 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:49.281178    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:49.532649 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.532580    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" event={"ID":"2305f190-18a0-46c6-a291-264be71b6d0d","Type":"ContainerStarted","Data":"635723327ba683983f1731f9979d1dd8510ab27a8152c8a97750b29b538e47c5"}
Jul 28 01:25:49.536483 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.536465    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:25:49.536869 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.536850    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" event={"ID":"69a3339f-758b-4280-98bc-1bde23005e7b","Type":"ContainerStarted","Data":"966ce6b218fbd796e71cf503df7147baa67bcbe173048c1c4dc14aae1dacf456"}
Jul 28 01:25:49.537045 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.537014    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:49.537151 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.537083    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:49.551842 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.551823    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:25:49.589049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.588996    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24" podStartSLOduration=9.144348909 podStartE2EDuration="29.588980134s" totalImagesPullingTime="20.444631225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:49.587801283 +0000 UTC m=+29.760546532" watchObservedRunningTime="2026-07-28 01:25:49.588980134 +0000 UTC m=+29.761725383"
Jul 28 01:25:49.592009 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.591989    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d8c8n"]
Jul 28 01:25:49.592096 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.592076    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:49.592201 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:49.592180    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:49.592731 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.592708    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hwr6q"]
Jul 28 01:25:49.592805 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.592783    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:49.592879 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:49.592850    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:49.593761 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.593739    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qd9t"]
Jul 28 01:25:49.593853 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:49.593827    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:49.593939 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:49.593908    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:50.541198 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:50.541017    2556 generic.go:356] "Generic (PLEG): container finished" podID="2305f190-18a0-46c6-a291-264be71b6d0d" containerID="635723327ba683983f1731f9979d1dd8510ab27a8152c8a97750b29b538e47c5" exitCode=0
Jul 28 01:25:50.541609 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:50.541111    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" event={"ID":"2305f190-18a0-46c6-a291-264be71b6d0d","Type":"ContainerDied","Data":"635723327ba683983f1731f9979d1dd8510ab27a8152c8a97750b29b538e47c5"}
Jul 28 01:25:51.281491 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:51.281441    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:51.281491 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:51.281486    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:51.281734 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:51.281453    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:51.281734 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:51.281587    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:51.281734 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:51.281633    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:51.281734 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:51.281702    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:52.548621 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:52.548583    2556 generic.go:356] "Generic (PLEG): container finished" podID="2305f190-18a0-46c6-a291-264be71b6d0d" containerID="a64830ad7dfa9af20facb83e811b850941b346a212e22b85ca3ea28517af2f7a" exitCode=0
Jul 28 01:25:52.549112 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:52.548642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" event={"ID":"2305f190-18a0-46c6-a291-264be71b6d0d","Type":"ContainerDied","Data":"a64830ad7dfa9af20facb83e811b850941b346a212e22b85ca3ea28517af2f7a"}
Jul 28 01:25:52.967031 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:52.966996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:52.967031 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:52.967036    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:52.967220 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:52.967128    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:52.967220 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:52.967144    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:52.967220 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:52.967166    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:52.967220 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:52.967175    2556 projected.go:196] Error preparing data for projected volume kube-api-access-48rfx for pod openshift-network-diagnostics/network-check-target-d8c8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:52.967220 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:52.967178    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret podName:ffc9bda4-ee09-499b-bb30-db6c295d9ec2 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:24.967164249 +0000 UTC m=+65.139909476 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret") pod "global-pull-secret-syncer-hwr6q" (UID: "ffc9bda4-ee09-499b-bb30-db6c295d9ec2") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:52.967220 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:52.967215    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx podName:68629e90-6236-451d-8615-a3fac3628ea0 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:24.967203409 +0000 UTC m=+65.139948635 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-48rfx" (UniqueName: "kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx") pod "network-check-target-d8c8n" (UID: "68629e90-6236-451d-8615-a3fac3628ea0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:53.068035 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:53.068001    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:53.068174 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:53.068147    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:53.068233 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:53.068217    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs podName:294ecef6-7cbe-446d-96f6-36de1ab923aa nodeName:}" failed. No retries permitted until 2026-07-28 01:26:25.06819977 +0000 UTC m=+65.240945001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs") pod "network-metrics-daemon-9qd9t" (UID: "294ecef6-7cbe-446d-96f6-36de1ab923aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:53.281549 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:53.281512    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:53.281549 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:53.281530    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:53.281549 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:53.281542    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:53.281836 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:53.281668    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qd9t" podUID="294ecef6-7cbe-446d-96f6-36de1ab923aa"
Jul 28 01:25:53.281836 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:53.281736    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8c8n" podUID="68629e90-6236-451d-8615-a3fac3628ea0"
Jul 28 01:25:53.281939 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:53.281847    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwr6q" podUID="ffc9bda4-ee09-499b-bb30-db6c295d9ec2"
Jul 28 01:25:54.606117 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.606036    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-37.ec2.internal" event="NodeReady"
Jul 28 01:25:54.606493 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.606200    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 01:25:54.669728 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.669692    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69b64567b4-c7jm7"]
Jul 28 01:25:54.672218 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.672191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.676883 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.676854    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 01:25:54.677049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.676901    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-knzvx\""
Jul 28 01:25:54.677049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.676996    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 01:25:54.677292 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.677275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 01:25:54.682243 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.682221    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-79nrr"]
Jul 28 01:25:54.684525 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.684504    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.688045 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.688020    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 01:25:54.688136 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.688103    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mh7n2\""
Jul 28 01:25:54.689395 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.689379    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 01:25:54.689865 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.689848    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x4n5c"]
Jul 28 01:25:54.692264 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.692244    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 01:25:54.692411 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.692388    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x4n5c"
Jul 28 01:25:54.695182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.695151    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 01:25:54.695523 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.695308    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 01:25:54.695523 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.695356    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 01:25:54.695675 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.695643    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h497g\""
Jul 28 01:25:54.711023 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.710997    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x4n5c"]
Jul 28 01:25:54.712452 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.712425    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69b64567b4-c7jm7"]
Jul 28 01:25:54.712569 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.712481    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-79nrr"]
Jul 28 01:25:54.780527 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780491    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-config-volume\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.780697 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780538    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.780697 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhfw6\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-kube-api-access-bhfw6\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.780795 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-tmp-dir\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.780795 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780739    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-bound-sa-token\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.780795 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780771    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zptp4\" (UniqueName: \"kubernetes.io/projected/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-kube-api-access-zptp4\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.780902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780822    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-registry-certificates\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.780902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780852    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a30566ed-1cc6-4db8-87f1-252244af5aef-cert\") pod \"ingress-canary-x4n5c\" (UID: \"a30566ed-1cc6-4db8-87f1-252244af5aef\") " pod="openshift-ingress-canary/ingress-canary-x4n5c"
Jul 28 01:25:54.780965 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780907    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpcx\" (UniqueName: \"kubernetes.io/projected/a30566ed-1cc6-4db8-87f1-252244af5aef-kube-api-access-kvpcx\") pod \"ingress-canary-x4n5c\" (UID: \"a30566ed-1cc6-4db8-87f1-252244af5aef\") " pod="openshift-ingress-canary/ingress-canary-x4n5c"
Jul 28 01:25:54.780994 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.780981    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2710cee-558f-444f-82a2-286ce35dbf62-ca-trust-extracted\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.781031 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.781009    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-trusted-ca\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.781072 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.781042    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.781120 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.781073    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-image-registry-private-configuration\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.781120 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.781100    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-installation-pull-secrets\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.881885 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.881800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2710cee-558f-444f-82a2-286ce35dbf62-ca-trust-extracted\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.881885 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.881845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-trusted-ca\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.882098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.881974    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.882098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882013    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-image-registry-private-configuration\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.882098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882037    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-installation-pull-secrets\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.882098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882063    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-config-volume\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.882098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882084    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.882346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfw6\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-kube-api-access-bhfw6\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.882346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882150    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-tmp-dir\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.882346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882175    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-bound-sa-token\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.882346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zptp4\" (UniqueName: \"kubernetes.io/projected/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-kube-api-access-zptp4\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.882346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882234    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-registry-certificates\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.882346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882247    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2710cee-558f-444f-82a2-286ce35dbf62-ca-trust-extracted\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.882346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882256    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a30566ed-1cc6-4db8-87f1-252244af5aef-cert\") pod \"ingress-canary-x4n5c\" (UID: \"a30566ed-1cc6-4db8-87f1-252244af5aef\") " pod="openshift-ingress-canary/ingress-canary-x4n5c"
Jul 28 01:25:54.882346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.882326    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpcx\" (UniqueName: \"kubernetes.io/projected/a30566ed-1cc6-4db8-87f1-252244af5aef-kube-api-access-kvpcx\") pod \"ingress-canary-x4n5c\" (UID: \"a30566ed-1cc6-4db8-87f1-252244af5aef\") " pod="openshift-ingress-canary/ingress-canary-x4n5c"
Jul 28 01:25:54.884524 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.884479    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-trusted-ca\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.885184 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:54.885160    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:25:54.885712 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:54.885631    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls podName:b87279e4-7169-4b8a-a1d7-bf1ff52a08ea nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.385599943 +0000 UTC m=+35.558345188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls") pod "dns-default-79nrr" (UID: "b87279e4-7169-4b8a-a1d7-bf1ff52a08ea") : secret "dns-default-metrics-tls" not found
Jul 28 01:25:54.885988 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:54.885972    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:25:54.886083 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:54.885990    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b64567b4-c7jm7: secret "image-registry-tls" not found
Jul 28 01:25:54.886083 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:54.886041    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls podName:b2710cee-558f-444f-82a2-286ce35dbf62 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.386025639 +0000 UTC m=+35.558770889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls") pod "image-registry-69b64567b4-c7jm7" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62") : secret "image-registry-tls" not found
Jul 28 01:25:54.886216 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.886188    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-config-volume\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.886462 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.886443    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-tmp-dir\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.887098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.887075    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-registry-certificates\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.888368 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.888337    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-image-registry-private-configuration\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.888446 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.888429    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-installation-pull-secrets\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.890384 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.890100    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a30566ed-1cc6-4db8-87f1-252244af5aef-cert\") pod \"ingress-canary-x4n5c\" (UID: \"a30566ed-1cc6-4db8-87f1-252244af5aef\") " pod="openshift-ingress-canary/ingress-canary-x4n5c"
Jul 28 01:25:54.892975 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.892948    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpcx\" (UniqueName: \"kubernetes.io/projected/a30566ed-1cc6-4db8-87f1-252244af5aef-kube-api-access-kvpcx\") pod \"ingress-canary-x4n5c\" (UID: \"a30566ed-1cc6-4db8-87f1-252244af5aef\") " pod="openshift-ingress-canary/ingress-canary-x4n5c"
Jul 28 01:25:54.893056 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.893027    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfw6\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-kube-api-access-bhfw6\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:54.895873 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.895851    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zptp4\" (UniqueName: \"kubernetes.io/projected/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-kube-api-access-zptp4\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:54.896155 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:54.896137    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-bound-sa-token\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:55.002277 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.002242    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x4n5c"
Jul 28 01:25:55.151445 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.151179    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x4n5c"]
Jul 28 01:25:55.280940 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.280900    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:25:55.281115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.280913    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:25:55.281115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.280921    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:25:55.284138 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.284109    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:25:55.284349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.284331    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:25:55.284600 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.284579    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4mq45\""
Jul 28 01:25:55.284730 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.284663    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:25:55.284730 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.284678    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:25:55.284730 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.284670    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5299b\""
Jul 28 01:25:55.387107 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.387069    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:55.387275 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:55.387123    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:55.387275 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:55.387241    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:25:55.387358 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:55.387275    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:25:55.387358 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:55.387294    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b64567b4-c7jm7: secret "image-registry-tls" not found
Jul 28 01:25:55.387358 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:55.387305    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls podName:b87279e4-7169-4b8a-a1d7-bf1ff52a08ea nodeName:}" failed. No retries permitted until 2026-07-28 01:25:56.387286174 +0000 UTC m=+36.560031416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls") pod "dns-default-79nrr" (UID: "b87279e4-7169-4b8a-a1d7-bf1ff52a08ea") : secret "dns-default-metrics-tls" not found
Jul 28 01:25:55.387488 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:55.387364    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls podName:b2710cee-558f-444f-82a2-286ce35dbf62 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:56.387343949 +0000 UTC m=+36.560089199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls") pod "image-registry-69b64567b4-c7jm7" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62") : secret "image-registry-tls" not found
Jul 28 01:25:56.396014 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:56.395976    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:56.396595 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:56.396117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:56.396595 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:56.396166    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:25:56.396595 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:56.396188    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b64567b4-c7jm7: secret "image-registry-tls" not found
Jul 28 01:25:56.396595 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:56.396205    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:25:56.396595 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:56.396255    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls podName:b2710cee-558f-444f-82a2-286ce35dbf62 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:58.396234953 +0000 UTC m=+38.568980193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls") pod "image-registry-69b64567b4-c7jm7" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62") : secret "image-registry-tls" not found
Jul 28 01:25:56.396595 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:56.396272    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls podName:b87279e4-7169-4b8a-a1d7-bf1ff52a08ea nodeName:}" failed. No retries permitted until 2026-07-28 01:25:58.396265685 +0000 UTC m=+38.569010912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls") pod "dns-default-79nrr" (UID: "b87279e4-7169-4b8a-a1d7-bf1ff52a08ea") : secret "dns-default-metrics-tls" not found
Jul 28 01:25:56.589264 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:25:56.589232    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30566ed_1cc6_4db8_87f1_252244af5aef.slice/crio-17a44ba19cecc79b6ccab8e7e7b23b591ae72e21e4c91784ee77536a1ccc42db WatchSource:0}: Error finding container 17a44ba19cecc79b6ccab8e7e7b23b591ae72e21e4c91784ee77536a1ccc42db: Status 404 returned error can't find the container with id 17a44ba19cecc79b6ccab8e7e7b23b591ae72e21e4c91784ee77536a1ccc42db
Jul 28 01:25:57.072690 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:57.072655    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:57.206288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:57.206254    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:57.287515 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:57.287454    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:57.563418 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:57.563333    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x4n5c" event={"ID":"a30566ed-1cc6-4db8-87f1-252244af5aef","Type":"ContainerStarted","Data":"17a44ba19cecc79b6ccab8e7e7b23b591ae72e21e4c91784ee77536a1ccc42db"}
Jul 28 01:25:57.566096 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:57.566070    2556 generic.go:356] "Generic (PLEG): container finished" podID="2305f190-18a0-46c6-a291-264be71b6d0d" containerID="455c2f26288dbd319675a203deb185571764f1eb094ee4e22afdda4342e7fe8e" exitCode=0
Jul 28 01:25:57.566210 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:57.566126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" event={"ID":"2305f190-18a0-46c6-a291-264be71b6d0d","Type":"ContainerDied","Data":"455c2f26288dbd319675a203deb185571764f1eb094ee4e22afdda4342e7fe8e"}
Jul 28 01:25:58.414280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:58.414247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:25:58.414446 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:58.414288    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:25:58.414446 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:58.414398    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:25:58.414446 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:58.414402    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:25:58.414544 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:58.414463    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls podName:b87279e4-7169-4b8a-a1d7-bf1ff52a08ea nodeName:}" failed. No retries permitted until 2026-07-28 01:26:02.414447359 +0000 UTC m=+42.587192589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls") pod "dns-default-79nrr" (UID: "b87279e4-7169-4b8a-a1d7-bf1ff52a08ea") : secret "dns-default-metrics-tls" not found
Jul 28 01:25:58.414544 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:58.414408    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b64567b4-c7jm7: secret "image-registry-tls" not found
Jul 28 01:25:58.414544 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:25:58.414533    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls podName:b2710cee-558f-444f-82a2-286ce35dbf62 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:02.414518955 +0000 UTC m=+42.587264184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls") pod "image-registry-69b64567b4-c7jm7" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62") : secret "image-registry-tls" not found
Jul 28 01:25:58.430769 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:58.430733    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:58.563810 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:58.563763    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:58.572043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:58.572003    2556 generic.go:356] "Generic (PLEG): container finished" podID="2305f190-18a0-46c6-a291-264be71b6d0d" containerID="9040807f1dbe90fac5528214598d0454cd52d8f326652d1b9e8d82661aaaae66" exitCode=0
Jul 28 01:25:58.572153 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:58.572077    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" event={"ID":"2305f190-18a0-46c6-a291-264be71b6d0d","Type":"ContainerDied","Data":"9040807f1dbe90fac5528214598d0454cd52d8f326652d1b9e8d82661aaaae66"}
Jul 28 01:25:58.666487 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:58.666381    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:59.096238 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:59.096211    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:59.174331 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:59.174300    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:59.577434 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:59.577402    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" event={"ID":"2305f190-18a0-46c6-a291-264be71b6d0d","Type":"ContainerStarted","Data":"b47dc4c75d9a360f487497a78f2f9912e84d9bb60c823213ef929ca4b5004abd"}
Jul 28 01:25:59.585007 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:59.584974    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:59.601117 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:59.601075    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4cdnz" podStartSLOduration=9.422299131 podStartE2EDuration="39.601059184s" totalImagesPullingTime="30.178760053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:59.599746923 +0000 UTC m=+39.772492172" watchObservedRunningTime="2026-07-28 01:25:59.601059184 +0000 UTC m=+39.773804438"
Jul 28 01:25:59.687284 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:59.687262    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:59.787158 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:59.787110    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:59.887466 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:25:59.887373    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:00.581765 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:00.581725    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x4n5c" event={"ID":"a30566ed-1cc6-4db8-87f1-252244af5aef","Type":"ContainerStarted","Data":"90005bb173ed468bac933e9fb6567fa4133637adc4379fbc3c695b052d6e2a45"}
Jul 28 01:26:00.600900 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:00.600852    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x4n5c" podStartSLOduration=3.5882434610000002 podStartE2EDuration="6.600835955s" totalImagesPullingTime="3.012592494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:00.59990827 +0000 UTC m=+40.772653519" watchObservedRunningTime="2026-07-28 01:26:00.600835955 +0000 UTC m=+40.773581203"
Jul 28 01:26:02.443362 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:02.443321    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:26:02.443362 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:02.443362    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:26:02.443745 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:02.443468    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:26:02.443745 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:02.443475    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:02.443745 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:02.443538    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls podName:b87279e4-7169-4b8a-a1d7-bf1ff52a08ea nodeName:}" failed. No retries permitted until 2026-07-28 01:26:10.443522437 +0000 UTC m=+50.616267683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls") pod "dns-default-79nrr" (UID: "b87279e4-7169-4b8a-a1d7-bf1ff52a08ea") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:02.443745 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:02.443479    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b64567b4-c7jm7: secret "image-registry-tls" not found
Jul 28 01:26:02.443745 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:02.443610    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls podName:b2710cee-558f-444f-82a2-286ce35dbf62 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:10.443598632 +0000 UTC m=+50.616343879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls") pod "image-registry-69b64567b4-c7jm7" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62") : secret "image-registry-tls" not found
Jul 28 01:26:08.215283 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:08.215247    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:08.316447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:08.316420    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:10.489246 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:10.489206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:26:10.489246 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:10.489248    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:26:10.489688 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:10.489364    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:10.489688 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:10.489433    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls podName:b87279e4-7169-4b8a-a1d7-bf1ff52a08ea nodeName:}" failed. No retries permitted until 2026-07-28 01:26:26.489417564 +0000 UTC m=+66.662162791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls") pod "dns-default-79nrr" (UID: "b87279e4-7169-4b8a-a1d7-bf1ff52a08ea") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:10.489688 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:10.489367    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:26:10.489688 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:10.489469    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b64567b4-c7jm7: secret "image-registry-tls" not found
Jul 28 01:26:10.489688 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:10.489517    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls podName:b2710cee-558f-444f-82a2-286ce35dbf62 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:26.489504138 +0000 UTC m=+66.662249366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls") pod "image-registry-69b64567b4-c7jm7" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62") : secret "image-registry-tls" not found
Jul 28 01:26:12.049507 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:12.049474    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:12.162301 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:12.162268    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:14.039483 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:14.039443    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:14.140042 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:14.140009    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:17.024337 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:17.024304    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:18.187026 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:18.186990    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:18.287592 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:18.287545    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:19.357754 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:19.357726    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:19.445337 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:19.445304    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:20.249188 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:20.249155    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:20.350242 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:20.350211    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:21.554719 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:21.554690    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrg24"
Jul 28 01:26:21.719479 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:21.719450    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:21.786577 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:21.786533    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:24.310367 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:24.310331    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:24.988568 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:24.988518    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:26:24.988773 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:24.988577    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:26:24.991434 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:24.991405    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:24.991434 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:24.991432    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:25.001255 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.001237    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:25.001490 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.001464    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffc9bda4-ee09-499b-bb30-db6c295d9ec2-original-pull-secret\") pod \"global-pull-secret-syncer-hwr6q\" (UID: \"ffc9bda4-ee09-499b-bb30-db6c295d9ec2\") " pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:26:25.011632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.011609    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rfx\" (UniqueName: \"kubernetes.io/projected/68629e90-6236-451d-8615-a3fac3628ea0-kube-api-access-48rfx\") pod \"network-check-target-d8c8n\" (UID: \"68629e90-6236-451d-8615-a3fac3628ea0\") " pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:26:25.089768 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.089738    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:26:25.092746 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.092727    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:25.100200 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:25.100181    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 01:26:25.100265 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:25.100241    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs podName:294ecef6-7cbe-446d-96f6-36de1ab923aa nodeName:}" failed. No retries permitted until 2026-07-28 01:27:29.100225009 +0000 UTC m=+129.272970235 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs") pod "network-metrics-daemon-9qd9t" (UID: "294ecef6-7cbe-446d-96f6-36de1ab923aa") : secret "metrics-daemon-secret" not found
Jul 28 01:26:25.294379 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.294308    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwr6q"
Jul 28 01:26:25.303058 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.303028    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4mq45\""
Jul 28 01:26:25.327678 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.327650    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:26:25.442632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.441589    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hwr6q"]
Jul 28 01:26:25.446982 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:26:25.446955    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc9bda4_ee09_499b_bb30_db6c295d9ec2.slice/crio-33bc50a3f96e175c98d6e559fa97113d1fc2dce788ee77d5b382768bbdcab4ff WatchSource:0}: Error finding container 33bc50a3f96e175c98d6e559fa97113d1fc2dce788ee77d5b382768bbdcab4ff: Status 404 returned error can't find the container with id 33bc50a3f96e175c98d6e559fa97113d1fc2dce788ee77d5b382768bbdcab4ff
Jul 28 01:26:25.455707 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.455686    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d8c8n"]
Jul 28 01:26:25.458404 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:26:25.458384    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68629e90_6236_451d_8615_a3fac3628ea0.slice/crio-5d27eca957e224fedb9830881f6b654e82fa820c6eb563d7d25e01cc22465083 WatchSource:0}: Error finding container 5d27eca957e224fedb9830881f6b654e82fa820c6eb563d7d25e01cc22465083: Status 404 returned error can't find the container with id 5d27eca957e224fedb9830881f6b654e82fa820c6eb563d7d25e01cc22465083
Jul 28 01:26:25.642892 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.642853    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hwr6q" event={"ID":"ffc9bda4-ee09-499b-bb30-db6c295d9ec2","Type":"ContainerStarted","Data":"33bc50a3f96e175c98d6e559fa97113d1fc2dce788ee77d5b382768bbdcab4ff"}
Jul 28 01:26:25.643989 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.643957    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d8c8n" event={"ID":"68629e90-6236-451d-8615-a3fac3628ea0","Type":"ContainerStarted","Data":"5d27eca957e224fedb9830881f6b654e82fa820c6eb563d7d25e01cc22465083"}
Jul 28 01:26:25.644648 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.644625    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:25.744450 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.744418    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:25.804361 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:25.804283    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:26.500532 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:26.500492    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:26:26.501012 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:26.500552    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:26:26.501012 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:26.500678    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:26.501012 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:26.500754    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls podName:b87279e4-7169-4b8a-a1d7-bf1ff52a08ea nodeName:}" failed. No retries permitted until 2026-07-28 01:26:58.500732262 +0000 UTC m=+98.673477507 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls") pod "dns-default-79nrr" (UID: "b87279e4-7169-4b8a-a1d7-bf1ff52a08ea") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:26.501012 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:26.500685    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 01:26:26.501012 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:26.500797    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b64567b4-c7jm7: secret "image-registry-tls" not found
Jul 28 01:26:26.501012 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:26.500874    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls podName:b2710cee-558f-444f-82a2-286ce35dbf62 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:58.500855995 +0000 UTC m=+98.673601234 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls") pod "image-registry-69b64567b4-c7jm7" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62") : secret "image-registry-tls" not found
Jul 28 01:26:28.091378 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:28.091246    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:28.153753 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:28.153721    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:28.964282 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:28.964252    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:29.065666 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:29.065636    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:30.658577 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:30.658516    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hwr6q" event={"ID":"ffc9bda4-ee09-499b-bb30-db6c295d9ec2","Type":"ContainerStarted","Data":"530131faceeb572d14e021935f883053b73943df8d73ad843553c4996ea95968"}
Jul 28 01:26:30.660024 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:30.659996    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d8c8n" event={"ID":"68629e90-6236-451d-8615-a3fac3628ea0","Type":"ContainerStarted","Data":"ee5eccae9de1132239b9d63ca0972a38492c7ab4954bd94e84a0f712025fbe6d"}
Jul 28 01:26:30.660136 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:30.660117    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:26:30.676011 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:30.675956    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hwr6q" podStartSLOduration=66.160860053 podStartE2EDuration="1m10.67593742s" totalImagesPullingTime="4.515077367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:30.675044994 +0000 UTC m=+70.847790244" watchObservedRunningTime="2026-07-28 01:26:30.67593742 +0000 UTC m=+70.848682670"
Jul 28 01:26:30.690637 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:30.690594    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d8c8n" podStartSLOduration=66.189442611 podStartE2EDuration="1m10.69057956s" totalImagesPullingTime="4.501136949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:30.690385018 +0000 UTC m=+70.863130267" watchObservedRunningTime="2026-07-28 01:26:30.69057956 +0000 UTC m=+70.863324800"
Jul 28 01:26:31.411258 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:31.411221    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:31.512347 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:31.512294    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:35.343499 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:35.343466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:35.411706 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:35.411682    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:35.933057 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:35.933026    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:37.846333 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:37.846306    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8"]
Jul 28 01:26:37.849309 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:37.849294    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8"
Jul 28 01:26:37.851953 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:37.851934    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fwt6h\""
Jul 28 01:26:37.853007 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:37.852991    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 01:26:37.853096 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:37.853017    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:37.860535 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:37.860516    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8"]
Jul 28 01:26:37.884278 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:37.884252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz2d\" (UniqueName: \"kubernetes.io/projected/bb2c5c28-a06f-4359-9aa2-5681f3731f8d-kube-api-access-7gz2d\") pod \"volume-data-source-validator-549c6d54cf-xblm8\" (UID: \"bb2c5c28-a06f-4359-9aa2-5681f3731f8d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8"
Jul 28 01:26:37.984811 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:37.984782    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz2d\" (UniqueName: \"kubernetes.io/projected/bb2c5c28-a06f-4359-9aa2-5681f3731f8d-kube-api-access-7gz2d\") pod \"volume-data-source-validator-549c6d54cf-xblm8\" (UID: \"bb2c5c28-a06f-4359-9aa2-5681f3731f8d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8"
Jul 28 01:26:37.998367 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:37.998340    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz2d\" (UniqueName: \"kubernetes.io/projected/bb2c5c28-a06f-4359-9aa2-5681f3731f8d-kube-api-access-7gz2d\") pod \"volume-data-source-validator-549c6d54cf-xblm8\" (UID: \"bb2c5c28-a06f-4359-9aa2-5681f3731f8d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8"
Jul 28 01:26:38.045383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:38.045343    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:38.144738 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:38.144657    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:38.158673 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:38.158656    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8"
Jul 28 01:26:38.280990 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:38.280957    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8"]
Jul 28 01:26:38.285011 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:26:38.284977    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2c5c28_a06f_4359_9aa2_5681f3731f8d.slice/crio-11a7201004bf5dca5ad32c1ad778aebf04c6a24d3696ee6b862bf3de95277c14 WatchSource:0}: Error finding container 11a7201004bf5dca5ad32c1ad778aebf04c6a24d3696ee6b862bf3de95277c14: Status 404 returned error can't find the container with id 11a7201004bf5dca5ad32c1ad778aebf04c6a24d3696ee6b862bf3de95277c14
Jul 28 01:26:38.679918 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:38.679884    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8" event={"ID":"bb2c5c28-a06f-4359-9aa2-5681f3731f8d","Type":"ContainerStarted","Data":"11a7201004bf5dca5ad32c1ad778aebf04c6a24d3696ee6b862bf3de95277c14"}
Jul 28 01:26:39.684070 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:39.683982    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8" event={"ID":"bb2c5c28-a06f-4359-9aa2-5681f3731f8d","Type":"ContainerStarted","Data":"a225618bf0e29d58182f2ae11f26b6d093f5dcda4ca769d373386bb55ff36bf7"}
Jul 28 01:26:39.700682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:39.700640    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-xblm8" podStartSLOduration=1.649505408 podStartE2EDuration="2.700625973s" totalImagesPullingTime="1.051120565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:39.699272401 +0000 UTC m=+79.872017652" watchObservedRunningTime="2026-07-28 01:26:39.700625973 +0000 UTC m=+79.873371236"
Jul 28 01:26:39.805166 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:39.805136    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:39.906539 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:39.906510    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.934474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:41.934419    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:42.027043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:42.027011    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:42.094712 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:42.094678    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:44.743878 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:44.743842    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:44.811398 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:44.811366    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.159365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.159325    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.226288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.226254    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.308262 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.308236    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.370533 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.370500    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.451166 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.451072    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.506523 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.506496    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dzw8g"]
Jul 28 01:26:45.510772 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.510754    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.515094 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.515077    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 01:26:45.515642 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.515617    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 01:26:45.515741 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.515714    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 01:26:45.516360 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.516340    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 01:26:45.516493 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.516378    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gqjq9\""
Jul 28 01:26:45.520411 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.520393    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dzw8g"]
Jul 28 01:26:45.541574 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.541525    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.639552 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.639520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7553c92a-8e6b-48a4-9ccc-a26280163ff1-data-volume\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.639708 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.639579    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8k4w\" (UniqueName: \"kubernetes.io/projected/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-api-access-b8k4w\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.639708 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.639662    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.639789 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.639730    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7553c92a-8e6b-48a4-9ccc-a26280163ff1-crio-socket\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.639789 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.639759    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.740445 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.740361    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7553c92a-8e6b-48a4-9ccc-a26280163ff1-data-volume\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.740445 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.740396    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8k4w\" (UniqueName: \"kubernetes.io/projected/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-api-access-b8k4w\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.740445 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.740418    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.740731 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.740454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7553c92a-8e6b-48a4-9ccc-a26280163ff1-crio-socket\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.740731 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.740471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.740731 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.740544    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7553c92a-8e6b-48a4-9ccc-a26280163ff1-crio-socket\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.740731 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:45.740549    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:45.740731 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:45.740654    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:46.240635102 +0000 UTC m=+86.413380329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:45.740731 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:45.740582    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:45.740731 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:45.740712    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:46.240700308 +0000 UTC m=+86.413445539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:45.741022 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.740772    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7553c92a-8e6b-48a4-9ccc-a26280163ff1-data-volume\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:45.748974 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:45.748954    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8k4w\" (UniqueName: \"kubernetes.io/projected/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-api-access-b8k4w\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:46.243821 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:46.243774    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:46.244002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:46.243856    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:46.244002 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:46.243925    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:46.244002 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:46.243930    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:46.244120 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:46.244003    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:47.243988072 +0000 UTC m=+87.416733300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:46.244120 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:46.244020    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:47.24401266 +0000 UTC m=+87.416757887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.165083 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:47.165044    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.250886 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:47.250861    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:47.251029 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:47.250924    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:47.251029 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:47.250984    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:47.251029 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:47.251007    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.251126 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:47.251044    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:49.251029472 +0000 UTC m=+89.423774699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:47.251126 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:47.251060    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:49.251051449 +0000 UTC m=+89.423796675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.281770 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:47.281743    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.479409 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:47.479349    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pjclb_cc5ba0f8-327b-4257-97da-632040885d84/dns-node-resolver/0.log"
Jul 28 01:26:48.327408 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:48.327358    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.482139 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:48.482118    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-st5g5_56abc673-f9ba-4b33-a82f-d0469ba37eb5/node-ca/0.log"
Jul 28 01:26:49.037633 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:49.037590    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:49.130899 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:49.130865    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:49.233182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:49.233149    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:49.266266 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:49.266239    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:49.266372 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:49.266288    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:49.266411 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:49.266374    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:49.266411 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:49.266380    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:49.266472 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:49.266423    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.266409743 +0000 UTC m=+93.439154970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:49.266472 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:49.266437    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.266430768 +0000 UTC m=+93.439175994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:49.280756 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:49.280732    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x4n5c_a30566ed-1cc6-4db8-87f1-252244af5aef/serve-healthcheck-canary/0.log"
Jul 28 01:26:49.331007 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:49.330980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:50.313041 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:50.313006    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:50.720240 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:50.720204    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:50.942159 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:50.942122    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:51.125215 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:51.125185    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:51.338127 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:51.338096    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:51.738313 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:51.738278    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.173974 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:52.173917    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.710324 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:52.710288    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.814008 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:52.813976    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.987984 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:52.987890    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.121447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:53.121404    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.285416 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:53.285326    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.296309 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:53.296279    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:53.296409 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:53.296362    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:26:53.296409 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:53.296399    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:53.296480 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:53.296464    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:53.296480 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:53.296476    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:01.296456199 +0000 UTC m=+101.469201426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:53.296549 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:26:53.296508    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls podName:7553c92a-8e6b-48a4-9ccc-a26280163ff1 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:01.296497963 +0000 UTC m=+101.469243206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dzw8g" (UID: "7553c92a-8e6b-48a4-9ccc-a26280163ff1") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:53.504321 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:53.504282    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.340391 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:54.340350    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.518326 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:54.518287    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.727163 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:54.727071    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.921873 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:54.921840    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.129274 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:55.129247    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.232574 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:55.232532    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.535587 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:55.535486    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.124822 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:56.124792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.321209 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:56.321159    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.941724 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:56.941688    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.300835 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:57.300761    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.707967 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:57.707933    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.895224 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:57.895188    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.294432 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.294396    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.532259 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.532231    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:26:58.532406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.532294    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:26:58.534634 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.534601    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b87279e4-7169-4b8a-a1d7-bf1ff52a08ea-metrics-tls\") pod \"dns-default-79nrr\" (UID: \"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea\") " pod="openshift-dns/dns-default-79nrr"
Jul 28 01:26:58.534634 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.534647    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"image-registry-69b64567b4-c7jm7\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") " pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:26:58.587874 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.587851    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-knzvx\""
Jul 28 01:26:58.594765 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.594740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:26:58.597674 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.597658    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mh7n2\""
Jul 28 01:26:58.604858 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.604838    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-79nrr"
Jul 28 01:26:58.735112 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.735086    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69b64567b4-c7jm7"]
Jul 28 01:26:58.737824 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:26:58.737800    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2710cee_558f_444f_82a2_286ce35dbf62.slice/crio-37dfd2b7f2a07ab0caf35d27b285045b2dd50041cd677b45b824080c00d4fe1d WatchSource:0}: Error finding container 37dfd2b7f2a07ab0caf35d27b285045b2dd50041cd677b45b824080c00d4fe1d: Status 404 returned error can't find the container with id 37dfd2b7f2a07ab0caf35d27b285045b2dd50041cd677b45b824080c00d4fe1d
Jul 28 01:26:58.751634 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.751609    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-79nrr"]
Jul 28 01:26:58.755248 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:26:58.755228    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87279e4_7169_4b8a_a1d7_bf1ff52a08ea.slice/crio-94abbaa3c3d67964126924ad0c8e91a7ae5b955149200e6e59f6f846b2d78375 WatchSource:0}: Error finding container 94abbaa3c3d67964126924ad0c8e91a7ae5b955149200e6e59f6f846b2d78375: Status 404 returned error can't find the container with id 94abbaa3c3d67964126924ad0c8e91a7ae5b955149200e6e59f6f846b2d78375
Jul 28 01:26:58.936188 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:58.936099    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:59.293621 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:59.293508    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:59.501110 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:59.501073    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:59.682666 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:59.682631    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:59.737874 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:59.737661    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-79nrr" event={"ID":"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea","Type":"ContainerStarted","Data":"94abbaa3c3d67964126924ad0c8e91a7ae5b955149200e6e59f6f846b2d78375"}
Jul 28 01:26:59.739418 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:59.739395    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7" event={"ID":"b2710cee-558f-444f-82a2-286ce35dbf62","Type":"ContainerStarted","Data":"71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5"}
Jul 28 01:26:59.739525 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:59.739460    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7" event={"ID":"b2710cee-558f-444f-82a2-286ce35dbf62","Type":"ContainerStarted","Data":"37dfd2b7f2a07ab0caf35d27b285045b2dd50041cd677b45b824080c00d4fe1d"}
Jul 28 01:26:59.739594 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:59.739546    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:26:59.829020 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:26:59.828965    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7" podStartSLOduration=99.828944851 podStartE2EDuration="1m39.828944851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:59.827299651 +0000 UTC m=+100.000044901" watchObservedRunningTime="2026-07-28 01:26:59.828944851 +0000 UTC m=+100.001690099"
Jul 28 01:27:00.251978 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:00.251945    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:00.743875 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:00.743833    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-79nrr" event={"ID":"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea","Type":"ContainerStarted","Data":"e36de67836a1ab091be0113b4357f550c41cb68466558f80cadb99e076ae78e1"}
Jul 28 01:27:00.744300 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:00.744022    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-79nrr"
Jul 28 01:27:00.744300 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:00.744058    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-79nrr" event={"ID":"b87279e4-7169-4b8a-a1d7-bf1ff52a08ea","Type":"ContainerStarted","Data":"bee8c4f43af8ad496518e75fff901882da7417f2a3e2bd51a4d7c402c951d692"}
Jul 28 01:27:00.765127 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:00.765050    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-79nrr" podStartSLOduration=65.138992047 podStartE2EDuration="1m6.765036973s" totalImagesPullingTime="1.626044926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:00.763611015 +0000 UTC m=+100.936356265" watchObservedRunningTime="2026-07-28 01:27:00.765036973 +0000 UTC m=+100.937782222"
Jul 28 01:27:01.355904 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.355860    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:27:01.356139 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.355926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:27:01.356404 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.356381    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7553c92a-8e6b-48a4-9ccc-a26280163ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:27:01.358348 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.358332    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7553c92a-8e6b-48a4-9ccc-a26280163ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dzw8g\" (UID: \"7553c92a-8e6b-48a4-9ccc-a26280163ff1\") " pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:27:01.419519 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.419457    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dzw8g"
Jul 28 01:27:01.564421 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.564393    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dzw8g"]
Jul 28 01:27:01.566645 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:01.566614    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7553c92a_8e6b_48a4_9ccc_a26280163ff1.slice/crio-deee908bd2f4e85af3901d00ba9db9bf4a9dcb157602e6ef875a5d1091a5b3a1 WatchSource:0}: Error finding container deee908bd2f4e85af3901d00ba9db9bf4a9dcb157602e6ef875a5d1091a5b3a1: Status 404 returned error can't find the container with id deee908bd2f4e85af3901d00ba9db9bf4a9dcb157602e6ef875a5d1091a5b3a1
Jul 28 01:27:01.639882 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.639801    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:01.664856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.664829    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d8c8n"
Jul 28 01:27:01.713084 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.713050    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:01.747444 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.747408    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzw8g" event={"ID":"7553c92a-8e6b-48a4-9ccc-a26280163ff1","Type":"ContainerStarted","Data":"deee908bd2f4e85af3901d00ba9db9bf4a9dcb157602e6ef875a5d1091a5b3a1"}
Jul 28 01:27:01.777087 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.777059    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:01.936277 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:01.936184    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:02.313169 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:02.313043    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:02.399818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:02.399786    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:02.715878 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:02.715843    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:02.780617 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:02.780580    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.329231 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:03.329196    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.552496 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:03.552454    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.652104 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:03.652007    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:04.137521 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:04.137435    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:04.208870 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:04.208837    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:06.767675 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:06.767648    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzw8g" event={"ID":"7553c92a-8e6b-48a4-9ccc-a26280163ff1","Type":"ContainerStarted","Data":"d48d00400eb9be3be8fb0f8f93b99b9905cc3022893c43caab5e1d843e88e602"}
Jul 28 01:27:08.213799 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:08.213759    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:08.313986 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:08.313893    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:09.654780 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.654741    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"]
Jul 28 01:27:09.656960 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.656935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"
Jul 28 01:27:09.667495 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.667469    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 01:27:09.667637 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.667613    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 01:27:09.676226 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.676198    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7qhsg\""
Jul 28 01:27:09.680043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.680020    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"]
Jul 28 01:27:09.683360 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.683339    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.726443 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.726417    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 01:27:09.727051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.727026    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 01:27:09.727181 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.727059    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 01:27:09.727181 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.727092    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 01:27:09.728214 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.728151    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"]
Jul 28 01:27:09.804448 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.804417    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"]
Jul 28 01:27:09.825340 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.825304    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-98g8r"]
Jul 28 01:27:09.829460 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.829431    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dc1ce8f6-c11b-4e90-89b3-356006dc583d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-q5n5g\" (UID: \"dc1ce8f6-c11b-4e90-89b3-356006dc583d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"
Jul 28 01:27:09.829611 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.829470    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9f4959c-bd57-4913-b7c7-034be4d590d7-tmp\") pod \"klusterlet-addon-workmgr-7689b949b-gr7p6\" (UID: \"c9f4959c-bd57-4913-b7c7-034be4d590d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.829611 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.829476    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-98g8r"
Jul 28 01:27:09.829611 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.829550    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jgz4\" (UniqueName: \"kubernetes.io/projected/c9f4959c-bd57-4913-b7c7-034be4d590d7-kube-api-access-6jgz4\") pod \"klusterlet-addon-workmgr-7689b949b-gr7p6\" (UID: \"c9f4959c-bd57-4913-b7c7-034be4d590d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.829611 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.829602    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc1ce8f6-c11b-4e90-89b3-356006dc583d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q5n5g\" (UID: \"dc1ce8f6-c11b-4e90-89b3-356006dc583d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"
Jul 28 01:27:09.829791 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.829639    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c9f4959c-bd57-4913-b7c7-034be4d590d7-klusterlet-config\") pod \"klusterlet-addon-workmgr-7689b949b-gr7p6\" (UID: \"c9f4959c-bd57-4913-b7c7-034be4d590d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.833018 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.832997    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-d9cr4\""
Jul 28 01:27:09.833565 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.833535    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 01:27:09.833678 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.833605    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 01:27:09.841796 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.841768    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-98g8r"]
Jul 28 01:27:09.928650 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.928578    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:09.930438 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.930411    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c9f4959c-bd57-4913-b7c7-034be4d590d7-klusterlet-config\") pod \"klusterlet-addon-workmgr-7689b949b-gr7p6\" (UID: \"c9f4959c-bd57-4913-b7c7-034be4d590d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.930544 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.930472    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dc1ce8f6-c11b-4e90-89b3-356006dc583d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-q5n5g\" (UID: \"dc1ce8f6-c11b-4e90-89b3-356006dc583d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"
Jul 28 01:27:09.930544 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.930502    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9f4959c-bd57-4913-b7c7-034be4d590d7-tmp\") pod \"klusterlet-addon-workmgr-7689b949b-gr7p6\" (UID: \"c9f4959c-bd57-4913-b7c7-034be4d590d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.930668 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.930569    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jgz4\" (UniqueName: \"kubernetes.io/projected/c9f4959c-bd57-4913-b7c7-034be4d590d7-kube-api-access-6jgz4\") pod \"klusterlet-addon-workmgr-7689b949b-gr7p6\" (UID: \"c9f4959c-bd57-4913-b7c7-034be4d590d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.930668 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.930602    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc1ce8f6-c11b-4e90-89b3-356006dc583d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q5n5g\" (UID: \"dc1ce8f6-c11b-4e90-89b3-356006dc583d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"
Jul 28 01:27:09.930668 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.930659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7vbn\" (UniqueName: \"kubernetes.io/projected/2789e7e3-91a1-447d-adc1-75c92a79ffa9-kube-api-access-x7vbn\") pod \"downloads-7c97dc4555-98g8r\" (UID: \"2789e7e3-91a1-447d-adc1-75c92a79ffa9\") " pod="openshift-console/downloads-7c97dc4555-98g8r"
Jul 28 01:27:09.930962 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.930940    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9f4959c-bd57-4913-b7c7-034be4d590d7-tmp\") pod \"klusterlet-addon-workmgr-7689b949b-gr7p6\" (UID: \"c9f4959c-bd57-4913-b7c7-034be4d590d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.931177 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.931159    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dc1ce8f6-c11b-4e90-89b3-356006dc583d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-q5n5g\" (UID: \"dc1ce8f6-c11b-4e90-89b3-356006dc583d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"
Jul 28 01:27:09.933217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.933193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc1ce8f6-c11b-4e90-89b3-356006dc583d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-q5n5g\" (UID: \"dc1ce8f6-c11b-4e90-89b3-356006dc583d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"
Jul 28 01:27:09.933307 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.933237    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c9f4959c-bd57-4913-b7c7-034be4d590d7-klusterlet-config\") pod \"klusterlet-addon-workmgr-7689b949b-gr7p6\" (UID: \"c9f4959c-bd57-4913-b7c7-034be4d590d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.940465 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.940445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jgz4\" (UniqueName: \"kubernetes.io/projected/c9f4959c-bd57-4913-b7c7-034be4d590d7-kube-api-access-6jgz4\") pod \"klusterlet-addon-workmgr-7689b949b-gr7p6\" (UID: \"c9f4959c-bd57-4913-b7c7-034be4d590d7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:09.966698 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.966678    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"
Jul 28 01:27:09.993116 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:09.993088    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:10.030007 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.029974    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:10.031062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.031042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7vbn\" (UniqueName: \"kubernetes.io/projected/2789e7e3-91a1-447d-adc1-75c92a79ffa9-kube-api-access-x7vbn\") pod \"downloads-7c97dc4555-98g8r\" (UID: \"2789e7e3-91a1-447d-adc1-75c92a79ffa9\") " pod="openshift-console/downloads-7c97dc4555-98g8r"
Jul 28 01:27:10.045967 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.045943    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7vbn\" (UniqueName: \"kubernetes.io/projected/2789e7e3-91a1-447d-adc1-75c92a79ffa9-kube-api-access-x7vbn\") pod \"downloads-7c97dc4555-98g8r\" (UID: \"2789e7e3-91a1-447d-adc1-75c92a79ffa9\") " pod="openshift-console/downloads-7c97dc4555-98g8r"
Jul 28 01:27:10.138839 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.138717    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-98g8r"
Jul 28 01:27:10.178055 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.177646    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g"]
Jul 28 01:27:10.180330 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:10.180257    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1ce8f6_c11b_4e90_89b3_356006dc583d.slice/crio-1e6fa4b115a51f7286f1310ccbe0b6588ba8da2aff4cf1eaea3372d9e6f9c96b WatchSource:0}: Error finding container 1e6fa4b115a51f7286f1310ccbe0b6588ba8da2aff4cf1eaea3372d9e6f9c96b: Status 404 returned error can't find the container with id 1e6fa4b115a51f7286f1310ccbe0b6588ba8da2aff4cf1eaea3372d9e6f9c96b
Jul 28 01:27:10.197617 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.197586    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"]
Jul 28 01:27:10.200752 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:10.200711    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f4959c_bd57_4913_b7c7_034be4d590d7.slice/crio-dc06d26d6c1e10dd2522d0b7ac6ea1dfbf60ed154487786e2afcf7043cbd73bc WatchSource:0}: Error finding container dc06d26d6c1e10dd2522d0b7ac6ea1dfbf60ed154487786e2afcf7043cbd73bc: Status 404 returned error can't find the container with id dc06d26d6c1e10dd2522d0b7ac6ea1dfbf60ed154487786e2afcf7043cbd73bc
Jul 28 01:27:10.273219 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.273188    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-98g8r"]
Jul 28 01:27:10.275833 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:10.275808    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2789e7e3_91a1_447d_adc1_75c92a79ffa9.slice/crio-5c1b12028da1ac729f10fbadacf7fb8105dfd06808aed335fa044ed65ae45366 WatchSource:0}: Error finding container 5c1b12028da1ac729f10fbadacf7fb8105dfd06808aed335fa044ed65ae45366: Status 404 returned error can't find the container with id 5c1b12028da1ac729f10fbadacf7fb8105dfd06808aed335fa044ed65ae45366
Jul 28 01:27:10.750508 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.750442    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-79nrr"
Jul 28 01:27:10.790737 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.790615    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6" event={"ID":"c9f4959c-bd57-4913-b7c7-034be4d590d7","Type":"ContainerStarted","Data":"dc06d26d6c1e10dd2522d0b7ac6ea1dfbf60ed154487786e2afcf7043cbd73bc"}
Jul 28 01:27:10.795799 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.795701    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzw8g" event={"ID":"7553c92a-8e6b-48a4-9ccc-a26280163ff1","Type":"ContainerStarted","Data":"5c11849232313e07b43ab95e955f27ee94ef334133e66d31fdcdb734c83d28ba"}
Jul 28 01:27:10.801686 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.801624    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-98g8r" event={"ID":"2789e7e3-91a1-447d-adc1-75c92a79ffa9","Type":"ContainerStarted","Data":"5c1b12028da1ac729f10fbadacf7fb8105dfd06808aed335fa044ed65ae45366"}
Jul 28 01:27:10.806131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:10.806020    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g" event={"ID":"dc1ce8f6-c11b-4e90-89b3-356006dc583d","Type":"ContainerStarted","Data":"1e6fa4b115a51f7286f1310ccbe0b6588ba8da2aff4cf1eaea3372d9e6f9c96b"}
Jul 28 01:27:11.465907 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:11.465810    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:11.535697 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:11.535643    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:11.814268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:11.813545    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g" event={"ID":"dc1ce8f6-c11b-4e90-89b3-356006dc583d","Type":"ContainerStarted","Data":"9082e6c7fbb5ed7dee200647fd56c6808ec5b3c28c9c80dc3fe343f76f17e3b6"}
Jul 28 01:27:11.846467 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:11.846405    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-q5n5g" podStartSLOduration=1.330065587 podStartE2EDuration="2.846383471s" totalImagesPullingTime="1.516317884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:11.846256568 +0000 UTC m=+112.019001817" watchObservedRunningTime="2026-07-28 01:27:11.846383471 +0000 UTC m=+112.019128722"
Jul 28 01:27:12.464720 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:12.464686    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.897343 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:12.897309    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.986963 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:12.986920    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:15.342263 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.342234    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bcc8879b6-8sscg"]
Jul 28 01:27:15.346294 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.346270    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.348954 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.348929    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 01:27:15.349108 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.349088    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 01:27:15.349189 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.349169    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 01:27:15.350288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.350270    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 01:27:15.350365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.350322    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dsjfs\""
Jul 28 01:27:15.350426 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.350273    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 01:27:15.356498 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.356468    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bcc8879b6-8sscg"]
Jul 28 01:27:15.476874 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.476826    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-config\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.477064 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.476878    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-oauth-config\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.477064 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.476910    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-serving-cert\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.477064 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.476954    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-service-ca\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.477064 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.477034    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqr4r\" (UniqueName: \"kubernetes.io/projected/c19807cf-e7da-493b-b41c-8f1f3d74610c-kube-api-access-pqr4r\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.477256 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.477068    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-oauth-serving-cert\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.577418 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.577382    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqr4r\" (UniqueName: \"kubernetes.io/projected/c19807cf-e7da-493b-b41c-8f1f3d74610c-kube-api-access-pqr4r\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.577633 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.577436    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-oauth-serving-cert\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.577633 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.577483    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-config\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.577633 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.577510    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-oauth-config\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.577633 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.577541    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-serving-cert\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.577633 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.577605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-service-ca\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.578222 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.578198    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-service-ca\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.578988 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.578960    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-oauth-serving-cert\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.579447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.579428    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-config\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.582362 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.582340    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-oauth-config\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.582793 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.582758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-serving-cert\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.587823 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.587783    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqr4r\" (UniqueName: \"kubernetes.io/projected/c19807cf-e7da-493b-b41c-8f1f3d74610c-kube-api-access-pqr4r\") pod \"console-bcc8879b6-8sscg\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") " pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.659822 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.659725    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:15.739820 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.739736    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:15.910893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.910762    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:15.917335 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.917287    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"]
Jul 28 01:27:15.919633 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.919604    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"
Jul 28 01:27:15.922250 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.922227    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 01:27:15.922372 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.922257    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4rppp\""
Jul 28 01:27:15.933281 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:15.933254    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"]
Jul 28 01:27:16.082051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.081914    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3303926-a6d6-4c36-9a9d-4f048950bc56-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wxf69\" (UID: \"b3303926-a6d6-4c36-9a9d-4f048950bc56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"
Jul 28 01:27:16.182489 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.182394    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3303926-a6d6-4c36-9a9d-4f048950bc56-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wxf69\" (UID: \"b3303926-a6d6-4c36-9a9d-4f048950bc56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"
Jul 28 01:27:16.182706 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:27:16.182626    2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 01:27:16.182781 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:27:16.182711    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3303926-a6d6-4c36-9a9d-4f048950bc56-tls-certificates podName:b3303926-a6d6-4c36-9a9d-4f048950bc56 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:16.682690991 +0000 UTC m=+116.855436223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b3303926-a6d6-4c36-9a9d-4f048950bc56-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-wxf69" (UID: "b3303926-a6d6-4c36-9a9d-4f048950bc56") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 01:27:16.505138 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.505070    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:16.687820 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.687723    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3303926-a6d6-4c36-9a9d-4f048950bc56-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wxf69\" (UID: \"b3303926-a6d6-4c36-9a9d-4f048950bc56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"
Jul 28 01:27:16.690910 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.690841    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3303926-a6d6-4c36-9a9d-4f048950bc56-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wxf69\" (UID: \"b3303926-a6d6-4c36-9a9d-4f048950bc56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"
Jul 28 01:27:16.733986 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.733913    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bcc8879b6-8sscg"]
Jul 28 01:27:16.738314 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:16.738277    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19807cf_e7da_493b_b41c_8f1f3d74610c.slice/crio-c382f35f1d672d246c7297549189f2a1d71a6e2a360830b8949a81824ece49ba WatchSource:0}: Error finding container c382f35f1d672d246c7297549189f2a1d71a6e2a360830b8949a81824ece49ba: Status 404 returned error can't find the container with id c382f35f1d672d246c7297549189f2a1d71a6e2a360830b8949a81824ece49ba
Jul 28 01:27:16.830951 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.830916    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"
Jul 28 01:27:16.835594 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.835541    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcc8879b6-8sscg" event={"ID":"c19807cf-e7da-493b-b41c-8f1f3d74610c","Type":"ContainerStarted","Data":"c382f35f1d672d246c7297549189f2a1d71a6e2a360830b8949a81824ece49ba"}
Jul 28 01:27:16.837149 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.837123    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6" event={"ID":"c9f4959c-bd57-4913-b7c7-034be4d590d7","Type":"ContainerStarted","Data":"036cfec1124758da7c8262063b35315b1d80dff8bda20bf29596a2310c97b489"}
Jul 28 01:27:16.837483 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.837456    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:16.839127 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.839107    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6"
Jul 28 01:27:16.839711 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.839690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dzw8g" event={"ID":"7553c92a-8e6b-48a4-9ccc-a26280163ff1","Type":"ContainerStarted","Data":"88fbfd302ca35ec6612c5e28cdad31457723dc1075b01385862c598f191edfe3"}
Jul 28 01:27:16.886573 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.886502    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689b949b-gr7p6" podStartSLOduration=1.524009955 podStartE2EDuration="7.886481525s" totalImagesPullingTime="6.36247157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.886463186 +0000 UTC m=+117.059208436" watchObservedRunningTime="2026-07-28 01:27:16.886481525 +0000 UTC m=+117.059226775"
Jul 28 01:27:16.943708 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.943659    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dzw8g" podStartSLOduration=17.088286091 podStartE2EDuration="31.943636718s" totalImagesPullingTime="14.855350627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.942617645 +0000 UTC m=+117.115362923" watchObservedRunningTime="2026-07-28 01:27:16.943636718 +0000 UTC m=+117.116381966"
Jul 28 01:27:16.990486 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:16.990438    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"]
Jul 28 01:27:16.994948 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:16.994918    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3303926_a6d6_4c36_9a9d_4f048950bc56.slice/crio-81ecdc2bb550cce3d42fd5241a1bae9b25e17403f5d70dae342b11c49eb8d7c5 WatchSource:0}: Error finding container 81ecdc2bb550cce3d42fd5241a1bae9b25e17403f5d70dae342b11c49eb8d7c5: Status 404 returned error can't find the container with id 81ecdc2bb550cce3d42fd5241a1bae9b25e17403f5d70dae342b11c49eb8d7c5
Jul 28 01:27:17.175371 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:17.175297    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:17.308845 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:17.308809    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:17.846257 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:17.845850    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69" event={"ID":"b3303926-a6d6-4c36-9a9d-4f048950bc56","Type":"ContainerStarted","Data":"81ecdc2bb550cce3d42fd5241a1bae9b25e17403f5d70dae342b11c49eb8d7c5"}
Jul 28 01:27:18.600602 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:18.600550    2556 patch_prober.go:28] interesting pod/image-registry-69b64567b4-c7jm7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 01:27:18.600770 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:18.600624    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7" podUID="b2710cee-558f-444f-82a2-286ce35dbf62" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:27:18.852677 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:18.852540    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69" event={"ID":"b3303926-a6d6-4c36-9a9d-4f048950bc56","Type":"ContainerStarted","Data":"3d4b50a151a8a63970b6f0ec2094a57064c52b53db8d6f39d25ce37c8f01290f"}
Jul 28 01:27:18.873414 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:18.873365    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69" podStartSLOduration=2.730655251 podStartE2EDuration="3.873346511s" totalImagesPullingTime="1.14269126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:18.873101337 +0000 UTC m=+119.045846587" watchObservedRunningTime="2026-07-28 01:27:18.873346511 +0000 UTC m=+119.046091761"
Jul 28 01:27:18.947962 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:18.947912    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:19.014884 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:19.014848    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:19.157492 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:19.157354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:19.230521 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:19.230488    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:19.856492 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:19.856454    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"
Jul 28 01:27:19.862520 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:19.862497    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wxf69"
Jul 28 01:27:19.950303 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:19.950270    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:20.749866 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.749836    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:27:20.862506 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.862468    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcc8879b6-8sscg" event={"ID":"c19807cf-e7da-493b-b41c-8f1f3d74610c","Type":"ContainerStarted","Data":"efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a"}
Jul 28 01:27:20.880361 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.880317    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bcc8879b6-8sscg" podStartSLOduration=2.750097349 podStartE2EDuration="5.880300588s" totalImagesPullingTime="3.130203239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:20.879701034 +0000 UTC m=+121.052446280" watchObservedRunningTime="2026-07-28 01:27:20.880300588 +0000 UTC m=+121.053045828"
Jul 28 01:27:20.943854 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.943745    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:20.978724 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.978694    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-klb4j"]
Jul 28 01:27:20.982378 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.982349    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:20.985288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.985266    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 01:27:20.985525 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.985509    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8kzr6\""
Jul 28 01:27:20.986620 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.986598    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 01:27:20.986754 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.986639    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 01:27:20.986965 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.986944    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 01:27:20.986965 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.986952    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 01:27:20.993323 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:20.993304    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-klb4j"]
Jul 28 01:27:21.026594 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.026505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/196cacb9-9694-4493-a8e6-e41db2ce721f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.026594 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.026543    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfl8\" (UniqueName: \"kubernetes.io/projected/196cacb9-9694-4493-a8e6-e41db2ce721f-kube-api-access-2kfl8\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.026773 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.026714    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/196cacb9-9694-4493-a8e6-e41db2ce721f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.026773 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.026758    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/196cacb9-9694-4493-a8e6-e41db2ce721f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.044677 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.044647    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:21.128082 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.128050    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/196cacb9-9694-4493-a8e6-e41db2ce721f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.128082 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.128084    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/196cacb9-9694-4493-a8e6-e41db2ce721f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.128298 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.128108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/196cacb9-9694-4493-a8e6-e41db2ce721f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.128298 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.128138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfl8\" (UniqueName: \"kubernetes.io/projected/196cacb9-9694-4493-a8e6-e41db2ce721f-kube-api-access-2kfl8\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.128405 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:27:21.128299    2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 01:27:21.128405 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:27:21.128371    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/196cacb9-9694-4493-a8e6-e41db2ce721f-prometheus-operator-tls podName:196cacb9-9694-4493-a8e6-e41db2ce721f nodeName:}" failed. No retries permitted until 2026-07-28 01:27:21.628351584 +0000 UTC m=+121.801096830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/196cacb9-9694-4493-a8e6-e41db2ce721f-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-klb4j" (UID: "196cacb9-9694-4493-a8e6-e41db2ce721f") : secret "prometheus-operator-tls" not found
Jul 28 01:27:21.128882 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.128856    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/196cacb9-9694-4493-a8e6-e41db2ce721f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.130828 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.130806    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/196cacb9-9694-4493-a8e6-e41db2ce721f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.137106 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.137079    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfl8\" (UniqueName: \"kubernetes.io/projected/196cacb9-9694-4493-a8e6-e41db2ce721f-kube-api-access-2kfl8\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.144807 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.144777    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:21.244498 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.244463    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:21.432338 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.432302    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:21.494968 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.494930    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:21.632465 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.632430    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/196cacb9-9694-4493-a8e6-e41db2ce721f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.635356 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.635330    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/196cacb9-9694-4493-a8e6-e41db2ce721f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-klb4j\" (UID: \"196cacb9-9694-4493-a8e6-e41db2ce721f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:21.894619 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:21.894591    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j"
Jul 28 01:27:23.131941 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.131906    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54b6c76676-m6kqc"]
Jul 28 01:27:23.134410 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.134389    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.143583 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.143544    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 01:27:23.151181 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.151162    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54b6c76676-m6kqc"]
Jul 28 01:27:23.244694 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.244667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-config\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.244865 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.244730    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-trusted-ca-bundle\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.244865 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.244769    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlwtq\" (UniqueName: \"kubernetes.io/projected/d9fb96d4-f63d-43b1-89de-39597f1f5235-kube-api-access-vlwtq\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.244865 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.244800    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-oauth-serving-cert\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.244996 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.244900    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-service-ca\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.244996 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.244965    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-oauth-config\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.245092 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.245007    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-serving-cert\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.346186 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.346154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-oauth-config\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.346186 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.346194    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-serving-cert\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.346406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.346215    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-config\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.346406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.346252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-trusted-ca-bundle\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.346406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.346289    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlwtq\" (UniqueName: \"kubernetes.io/projected/d9fb96d4-f63d-43b1-89de-39597f1f5235-kube-api-access-vlwtq\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.346406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.346317    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-oauth-serving-cert\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.346406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.346360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-service-ca\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.347338 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.347309    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-config\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.347338 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.347322    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-service-ca\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.347540 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.347383    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-oauth-serving-cert\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.347540 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.347437    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-trusted-ca-bundle\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.349057 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.349036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-serving-cert\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.349148 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.349124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-oauth-config\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.369134 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.369107    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlwtq\" (UniqueName: \"kubernetes.io/projected/d9fb96d4-f63d-43b1-89de-39597f1f5235-kube-api-access-vlwtq\") pod \"console-54b6c76676-m6kqc\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") " pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:23.445612 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:23.445508    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:24.557552 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:24.557520    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:24.662012 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:24.661926    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:25.660392 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:25.660359    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:25.660838 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:25.660473    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:25.665906 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:25.665885    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:25.883062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:25.883031    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:27.953405 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:27.953370    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:28.053586 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:28.053533    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:28.114474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:28.114441    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:28.729588 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:28.729547    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54b6c76676-m6kqc"]
Jul 28 01:27:28.732076 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:28.732049    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9fb96d4_f63d_43b1_89de_39597f1f5235.slice/crio-862111c4f58435b9588b56c275ae65d10517b56592fe55b67c1c104d48bf4dfd WatchSource:0}: Error finding container 862111c4f58435b9588b56c275ae65d10517b56592fe55b67c1c104d48bf4dfd: Status 404 returned error can't find the container with id 862111c4f58435b9588b56c275ae65d10517b56592fe55b67c1c104d48bf4dfd
Jul 28 01:27:28.748588 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:28.748545    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-klb4j"]
Jul 28 01:27:28.751069 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:28.751036    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196cacb9_9694_4493_a8e6_e41db2ce721f.slice/crio-d3f37bac47318ec8d755701428a494f69c47c9891e821b19ea4e94ce4d08545d WatchSource:0}: Error finding container d3f37bac47318ec8d755701428a494f69c47c9891e821b19ea4e94ce4d08545d: Status 404 returned error can't find the container with id d3f37bac47318ec8d755701428a494f69c47c9891e821b19ea4e94ce4d08545d
Jul 28 01:27:28.889978 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:28.889894    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j" event={"ID":"196cacb9-9694-4493-a8e6-e41db2ce721f","Type":"ContainerStarted","Data":"d3f37bac47318ec8d755701428a494f69c47c9891e821b19ea4e94ce4d08545d"}
Jul 28 01:27:28.890957 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:28.890935    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b6c76676-m6kqc" event={"ID":"d9fb96d4-f63d-43b1-89de-39597f1f5235","Type":"ContainerStarted","Data":"862111c4f58435b9588b56c275ae65d10517b56592fe55b67c1c104d48bf4dfd"}
Jul 28 01:27:28.958879 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:28.958853    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:29.060864 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.060823    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:29.196245 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.196170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:27:29.198677 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.198649    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294ecef6-7cbe-446d-96f6-36de1ab923aa-metrics-certs\") pod \"network-metrics-daemon-9qd9t\" (UID: \"294ecef6-7cbe-446d-96f6-36de1ab923aa\") " pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:27:29.209328 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.209307    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5299b\""
Jul 28 01:27:29.216729 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.216703    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qd9t"
Jul 28 01:27:29.374996 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.374890    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qd9t"]
Jul 28 01:27:29.377310 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:29.377283    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294ecef6_7cbe_446d_96f6_36de1ab923aa.slice/crio-c942db12fa0439e19551466cf8d14aa64fc436f22d1eae8362215485af49e5d8 WatchSource:0}: Error finding container c942db12fa0439e19551466cf8d14aa64fc436f22d1eae8362215485af49e5d8: Status 404 returned error can't find the container with id c942db12fa0439e19551466cf8d14aa64fc436f22d1eae8362215485af49e5d8
Jul 28 01:27:29.900257 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.900092    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qd9t" event={"ID":"294ecef6-7cbe-446d-96f6-36de1ab923aa","Type":"ContainerStarted","Data":"c942db12fa0439e19551466cf8d14aa64fc436f22d1eae8362215485af49e5d8"}
Jul 28 01:27:29.904634 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.904258    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b6c76676-m6kqc" event={"ID":"d9fb96d4-f63d-43b1-89de-39597f1f5235","Type":"ContainerStarted","Data":"84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7"}
Jul 28 01:27:29.909767 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.909736    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-98g8r" event={"ID":"2789e7e3-91a1-447d-adc1-75c92a79ffa9","Type":"ContainerStarted","Data":"e6ac76c57611efa087b63fd09de3af75d99ffb9e666bce6dfd19e39f7baa91c0"}
Jul 28 01:27:29.910260 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.910192    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-98g8r"
Jul 28 01:27:29.925785 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.925688    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-98g8r"
Jul 28 01:27:29.929452 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.927636    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54b6c76676-m6kqc" podStartSLOduration=6.927615663 podStartE2EDuration="6.927615663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:29.926727761 +0000 UTC m=+130.099473011" watchObservedRunningTime="2026-07-28 01:27:29.927615663 +0000 UTC m=+130.100360913"
Jul 28 01:27:29.946298 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:29.946221    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-98g8r" podStartSLOduration=1.989541147 podStartE2EDuration="20.946200769s" totalImagesPullingTime="18.956659622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:29.945683008 +0000 UTC m=+130.118428270" watchObservedRunningTime="2026-07-28 01:27:29.946200769 +0000 UTC m=+130.118946018"
Jul 28 01:27:31.729064 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:31.729025    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:31.921193 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:31.920889    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j" event={"ID":"196cacb9-9694-4493-a8e6-e41db2ce721f","Type":"ContainerStarted","Data":"68d5229ea0ba0be89e6bb219ae00a26ca47fa172c5914e19fd4f0a7ae0ca437e"}
Jul 28 01:27:31.921193 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:31.920957    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j" event={"ID":"196cacb9-9694-4493-a8e6-e41db2ce721f","Type":"ContainerStarted","Data":"a79d61e15ba4ec4b39accbdf150aa3595103d8469c34ca0285727e16a3056198"}
Jul 28 01:27:31.923116 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:31.923082    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qd9t" event={"ID":"294ecef6-7cbe-446d-96f6-36de1ab923aa","Type":"ContainerStarted","Data":"4c19d7a84228d81fbc98eec627dca1b82241043cc27c55eba2d698d273f955c2"}
Jul 28 01:27:31.923269 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:31.923139    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qd9t" event={"ID":"294ecef6-7cbe-446d-96f6-36de1ab923aa","Type":"ContainerStarted","Data":"8fa38a0a848693b942bd54a1ce35ed12fd91bf5f1fe51e5573e7bc987aeb3e2c"}
Jul 28 01:27:31.940953 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:31.940900    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-klb4j" podStartSLOduration=9.408053222 podStartE2EDuration="11.94088489s" totalImagesPullingTime="2.532831668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:31.938509199 +0000 UTC m=+132.111254446" watchObservedRunningTime="2026-07-28 01:27:31.94088489 +0000 UTC m=+132.113630138"
Jul 28 01:27:31.961753 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:31.961694    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9qd9t" podStartSLOduration=130.052280496 podStartE2EDuration="2m11.961675776s" totalImagesPullingTime="1.90939528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:31.961249698 +0000 UTC m=+132.133994947" watchObservedRunningTime="2026-07-28 01:27:31.961675776 +0000 UTC m=+132.134421024"
Jul 28 01:27:32.043548 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:32.043510    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:32.144902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:32.144864    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:32.217755 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:32.217668    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:33.202285 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.202249    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69b64567b4-c7jm7"]
Jul 28 01:27:33.295617 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.295581    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:33.368287 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.368248    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:33.446519 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.446482    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:33.446703 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.446597    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:33.452182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.452160    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:33.502100 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.502015    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:33.567059 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.567023    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:33.935962 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.935934    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:27:33.987349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:33.987311    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bcc8879b6-8sscg"]
Jul 28 01:27:34.144077 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.144038    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:34.242443 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.242368    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:34.348294 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.348256    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:34.420409 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.420377    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"]
Jul 28 01:27:34.460288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.460260    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tfknf"]
Jul 28 01:27:34.460456 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.460406    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.463487 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.463465    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x4c4d\""
Jul 28 01:27:34.463713 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.463696    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 01:27:34.464549 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.464516    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 01:27:34.464828 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.464811    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 01:27:34.481307 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.481284    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"]
Jul 28 01:27:34.481441 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.481423    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.485058 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.485030    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vt9th\""
Jul 28 01:27:34.485165 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.485145    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 01:27:34.485966 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.485946    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 01:27:34.486523 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.486501    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 01:27:34.486851 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.486828    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:34.538690 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538611    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b31ad112-13c2-4435-a57e-008902a2c744-root\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.538690 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538654    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/60477cfa-b522-48b8-9280-c3528879b2e0-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.538690 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538693    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-textfile\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.538913 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60477cfa-b522-48b8-9280-c3528879b2e0-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.538913 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.538913 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538771    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b31ad112-13c2-4435-a57e-008902a2c744-sys\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.538913 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538801    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-wtmp\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.538913 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538888    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/60477cfa-b522-48b8-9280-c3528879b2e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.539123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538917    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60477cfa-b522-48b8-9280-c3528879b2e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.539123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538947    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b31ad112-13c2-4435-a57e-008902a2c744-metrics-client-ca\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.539123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538972    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5p6b\" (UniqueName: \"kubernetes.io/projected/60477cfa-b522-48b8-9280-c3528879b2e0-kube-api-access-f5p6b\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.539123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.538998    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-tls\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.539123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.539041    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg42x\" (UniqueName: \"kubernetes.io/projected/b31ad112-13c2-4435-a57e-008902a2c744-kube-api-access-kg42x\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.539123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.539071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60477cfa-b522-48b8-9280-c3528879b2e0-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.539123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.539112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-accelerators-collector-config\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.631580 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.631508    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:34.639780 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.639745    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b31ad112-13c2-4435-a57e-008902a2c744-root\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.639930 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.639786    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/60477cfa-b522-48b8-9280-c3528879b2e0-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.639930 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.639869    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b31ad112-13c2-4435-a57e-008902a2c744-root\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.639930 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.639920    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-textfile\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.640090 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.639953    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60477cfa-b522-48b8-9280-c3528879b2e0-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.640090 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.639988    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.640090 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640018    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b31ad112-13c2-4435-a57e-008902a2c744-sys\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.640090 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640065    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-wtmp\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.640268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640123    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/60477cfa-b522-48b8-9280-c3528879b2e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.640268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640146    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/60477cfa-b522-48b8-9280-c3528879b2e0-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.640268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640159    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60477cfa-b522-48b8-9280-c3528879b2e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.640268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640194    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b31ad112-13c2-4435-a57e-008902a2c744-metrics-client-ca\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.640268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640221    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5p6b\" (UniqueName: \"kubernetes.io/projected/60477cfa-b522-48b8-9280-c3528879b2e0-kube-api-access-f5p6b\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.640268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640244    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-tls\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.640268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640270    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg42x\" (UniqueName: \"kubernetes.io/projected/b31ad112-13c2-4435-a57e-008902a2c744-kube-api-access-kg42x\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.640552 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640285    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60477cfa-b522-48b8-9280-c3528879b2e0-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.640552 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640307    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-accelerators-collector-config\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.640963 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640942    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-accelerators-collector-config\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.641036 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.640968    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b31ad112-13c2-4435-a57e-008902a2c744-metrics-client-ca\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.641036 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.641020    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b31ad112-13c2-4435-a57e-008902a2c744-sys\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.641177 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.641149    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-wtmp\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.641298 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.641207    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-textfile\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.642301 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.641814    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/60477cfa-b522-48b8-9280-c3528879b2e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.642683 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.642597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60477cfa-b522-48b8-9280-c3528879b2e0-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.643093 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.643066    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.643919 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.643896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b31ad112-13c2-4435-a57e-008902a2c744-node-exporter-tls\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.644408 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.644381    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60477cfa-b522-48b8-9280-c3528879b2e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.644527 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.644504    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60477cfa-b522-48b8-9280-c3528879b2e0-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.651644 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.651622    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5p6b\" (UniqueName: \"kubernetes.io/projected/60477cfa-b522-48b8-9280-c3528879b2e0-kube-api-access-f5p6b\") pod \"kube-state-metrics-56d9f944b4-xr9dr\" (UID: \"60477cfa-b522-48b8-9280-c3528879b2e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.652293 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.652263    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg42x\" (UniqueName: \"kubernetes.io/projected/b31ad112-13c2-4435-a57e-008902a2c744-kube-api-access-kg42x\") pod \"node-exporter-tfknf\" (UID: \"b31ad112-13c2-4435-a57e-008902a2c744\") " pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.723154 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.723121    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:34.771586 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.771539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"
Jul 28 01:27:34.791793 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.791726    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tfknf"
Jul 28 01:27:34.885507 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.885471    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:34.937975 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.937894    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tfknf" event={"ID":"b31ad112-13c2-4435-a57e-008902a2c744","Type":"ContainerStarted","Data":"46da28cca8b1331d836a3525d184cd0d9888d02bff1947f3896226ba5f36c456"}
Jul 28 01:27:34.965940 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:34.965881    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr"]
Jul 28 01:27:34.968617 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:34.968587    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60477cfa_b522_48b8_9280_c3528879b2e0.slice/crio-a2ee25680870a22b29d2431866dbb104a4eae42f8d781f008cc0f3cabb048e1a WatchSource:0}: Error finding container a2ee25680870a22b29d2431866dbb104a4eae42f8d781f008cc0f3cabb048e1a: Status 404 returned error can't find the container with id a2ee25680870a22b29d2431866dbb104a4eae42f8d781f008cc0f3cabb048e1a
Jul 28 01:27:35.114050 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:35.114014    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:35.388212 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:35.388048    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:35.491740 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:35.491708    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:35.588412 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:35.588287    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:35.944615 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:35.944499    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr" event={"ID":"60477cfa-b522-48b8-9280-c3528879b2e0","Type":"ContainerStarted","Data":"a2ee25680870a22b29d2431866dbb104a4eae42f8d781f008cc0f3cabb048e1a"}
Jul 28 01:27:37.361518 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.361414    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.421161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.421120    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5988bb7987-nsstg"]
Jul 28 01:27:37.446625 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.446591    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5988bb7987-nsstg"]
Jul 28 01:27:37.446831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.446688    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.449870 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.449847    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 01:27:37.450264 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.450242    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8kskm\""
Jul 28 01:27:37.450451 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.450432    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 01:27:37.450543 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.450524    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 01:27:37.450682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.450662    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aa9m47jc6h70g\""
Jul 28 01:27:37.450775 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.450735    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 01:27:37.450905 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.450892    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 01:27:37.455051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.454985    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.567103 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.567015    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gkt4\" (UniqueName: \"kubernetes.io/projected/e1ce6711-b7fb-4482-a04d-3b4747e45c99-kube-api-access-5gkt4\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.567103 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.567097    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.567393 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.567135    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.567393 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.567158    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce6711-b7fb-4482-a04d-3b4747e45c99-metrics-client-ca\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.567393 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.567180    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-grpc-tls\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.567393 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.567206    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.567393 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.567232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-tls\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.567393 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.567268    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.607236 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.607195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.667947 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.667907    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gkt4\" (UniqueName: \"kubernetes.io/projected/e1ce6711-b7fb-4482-a04d-3b4747e45c99-kube-api-access-5gkt4\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.668131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.667980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.668131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.668015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.668131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.668042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce6711-b7fb-4482-a04d-3b4747e45c99-metrics-client-ca\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.668131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.668066    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-grpc-tls\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.668131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.668093    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.668131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.668119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-tls\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.668427 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.668150    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.669254 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.668970    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce6711-b7fb-4482-a04d-3b4747e45c99-metrics-client-ca\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.671614 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.671584    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.671832 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.671612    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-tls\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.671832 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.671811    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-grpc-tls\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.672096 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.672070    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.672234 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.672173    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.672304 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.672291    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1ce6711-b7fb-4482-a04d-3b4747e45c99-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.677497 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.677477    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gkt4\" (UniqueName: \"kubernetes.io/projected/e1ce6711-b7fb-4482-a04d-3b4747e45c99-kube-api-access-5gkt4\") pod \"thanos-querier-5988bb7987-nsstg\" (UID: \"e1ce6711-b7fb-4482-a04d-3b4747e45c99\") " pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.730620 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.730581    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.759524 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.759494    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:37.925054 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.925001    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.943775 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.943755    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5988bb7987-nsstg"]
Jul 28 01:27:37.945953 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:37.945926    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ce6711_b7fb_4482_a04d_3b4747e45c99.slice/crio-0aba30d1c168d65c8f77c99a599940b826cc78010f40f6cbec550289442cebed WatchSource:0}: Error finding container 0aba30d1c168d65c8f77c99a599940b826cc78010f40f6cbec550289442cebed: Status 404 returned error can't find the container with id 0aba30d1c168d65c8f77c99a599940b826cc78010f40f6cbec550289442cebed
Jul 28 01:27:37.954766 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.954737    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg" event={"ID":"e1ce6711-b7fb-4482-a04d-3b4747e45c99","Type":"ContainerStarted","Data":"0aba30d1c168d65c8f77c99a599940b826cc78010f40f6cbec550289442cebed"}
Jul 28 01:27:37.956594 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.956543    2556 generic.go:356] "Generic (PLEG): container finished" podID="b31ad112-13c2-4435-a57e-008902a2c744" containerID="8f6d0206c07cc029f192c3ae0cfe7f33a3b0f570a68d8fb8a085e3b5a8acfb24" exitCode=0
Jul 28 01:27:37.956750 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.956605    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tfknf" event={"ID":"b31ad112-13c2-4435-a57e-008902a2c744","Type":"ContainerDied","Data":"8f6d0206c07cc029f192c3ae0cfe7f33a3b0f570a68d8fb8a085e3b5a8acfb24"}
Jul 28 01:27:37.959384 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.959355    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr" event={"ID":"60477cfa-b522-48b8-9280-c3528879b2e0","Type":"ContainerStarted","Data":"e1c41ddb6c139c3b83775ff931bff3055fae4d4a97c984ebc2bc2e7f65cb09b2"}
Jul 28 01:27:37.959469 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.959417    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr" event={"ID":"60477cfa-b522-48b8-9280-c3528879b2e0","Type":"ContainerStarted","Data":"2e0491025ff679666a8cfebffb695cef5af425546b2c5a68a90940696c62a068"}
Jul 28 01:27:37.959469 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:37.959445    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr" event={"ID":"60477cfa-b522-48b8-9280-c3528879b2e0","Type":"ContainerStarted","Data":"7088b0fdc5a4a7792cf905fd2aca3ccc43a419dea09eb3932ac80617092ae5b8"}
Jul 28 01:27:38.002488 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:38.002445    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xr9dr" podStartSLOduration=1.977485512 podStartE2EDuration="4.002431519s" totalImagesPullingTime="2.024946007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:38.00064249 +0000 UTC m=+138.173387739" watchObservedRunningTime="2026-07-28 01:27:38.002431519 +0000 UTC m=+138.175176782"
Jul 28 01:27:38.007885 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:38.007859    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.212928 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:38.212894    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.314477 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:38.314447    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:38.972703 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:38.972666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tfknf" event={"ID":"b31ad112-13c2-4435-a57e-008902a2c744","Type":"ContainerStarted","Data":"a7df40ad72bed3175037044727eb6ffec0ad37c7325c7e5baf5e71ce9e05726f"}
Jul 28 01:27:38.973262 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:38.972762    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tfknf" event={"ID":"b31ad112-13c2-4435-a57e-008902a2c744","Type":"ContainerStarted","Data":"6750065d6f17c7eda1e72a83effd8b274edb953c4b9f092345a75322b1a3b958"}
Jul 28 01:27:38.995990 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:38.995939    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tfknf" podStartSLOduration=2.716467394 podStartE2EDuration="4.995920649s" totalImagesPullingTime="2.156044815s" totalInitContainerRuntime="123.40844ms" isStatefulPod=true podCreationTimestamp="2026-07-28 01:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:38.994242572 +0000 UTC m=+139.166987843" watchObservedRunningTime="2026-07-28 01:27:38.995920649 +0000 UTC m=+139.168665898"
Jul 28 01:27:39.149672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.149631    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:39.149897 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.149806    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-95c7f4b44-2t84p"]
Jul 28 01:27:39.164341 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.164322    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.186869 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.186849    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95c7f4b44-2t84p"]
Jul 28 01:27:39.281667 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.281577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-service-ca\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.281667 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.281626    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-trusted-ca-bundle\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.281855 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.281739    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-serving-cert\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.281855 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.281789    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-oauth-config\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.281931 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.281856    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-console-config\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.281931 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.281875    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhndc\" (UniqueName: \"kubernetes.io/projected/ddf8af51-d279-41e8-a9ed-97b371159291-kube-api-access-zhndc\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.281931 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.281898    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-oauth-serving-cert\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.301753 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.301708    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:39.383233 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.383204    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-service-ca\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.383418 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.383240    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-trusted-ca-bundle\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.383418 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.383281    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-serving-cert\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.383535 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.383415    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-oauth-config\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.383535 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.383490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-console-config\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.383535 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.383517    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhndc\" (UniqueName: \"kubernetes.io/projected/ddf8af51-d279-41e8-a9ed-97b371159291-kube-api-access-zhndc\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.383708 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.383549    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-oauth-serving-cert\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.384193 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.384087    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-service-ca\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.384309 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.384213    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-trusted-ca-bundle\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.384381 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.384358    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-oauth-serving-cert\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.384868 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.384836    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-console-config\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.386305 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.386277    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-oauth-config\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.386756 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.386740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-serving-cert\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.396053 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.396020    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhndc\" (UniqueName: \"kubernetes.io/projected/ddf8af51-d279-41e8-a9ed-97b371159291-kube-api-access-zhndc\") pod \"console-95c7f4b44-2t84p\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") " pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.410449 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.410425    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:39.473427 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.473395    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:39.605979 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.605942    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:39.879075 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.879051    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95c7f4b44-2t84p"]
Jul 28 01:27:39.881525 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:39.881500    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf8af51_d279_41e8_a9ed_97b371159291.slice/crio-512be241cb1e43751d7e5b0a92b76ade333b131d56f169979d32ec45633bbcab WatchSource:0}: Error finding container 512be241cb1e43751d7e5b0a92b76ade333b131d56f169979d32ec45633bbcab: Status 404 returned error can't find the container with id 512be241cb1e43751d7e5b0a92b76ade333b131d56f169979d32ec45633bbcab
Jul 28 01:27:39.979365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.979336    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c7f4b44-2t84p" event={"ID":"ddf8af51-d279-41e8-a9ed-97b371159291","Type":"ContainerStarted","Data":"e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184"}
Jul 28 01:27:39.979679 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:39.979392    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c7f4b44-2t84p" event={"ID":"ddf8af51-d279-41e8-a9ed-97b371159291","Type":"ContainerStarted","Data":"512be241cb1e43751d7e5b0a92b76ade333b131d56f169979d32ec45633bbcab"}
Jul 28 01:27:40.683091 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.683049    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:27:40.696505 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.696469    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.699975 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.699945    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 01:27:40.700135 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.700114    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 01:27:40.700198 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.700176    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 01:27:40.700334 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.700318    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k94qh\""
Jul 28 01:27:40.700408 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.700389    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 01:27:40.700987 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.700949    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 01:27:40.701091 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.701048    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 01:27:40.701482 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.700966    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 01:27:40.701677 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.701655    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 01:27:40.701771 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.701754    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 01:27:40.704015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.702359    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 01:27:40.704015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.702429    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 01:27:40.704015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.702691    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 01:27:40.704015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.702845    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 01:27:40.704015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.702884    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 01:27:40.704015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.702932    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dnjho185hojsd\""
Jul 28 01:27:40.705805 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.705271    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 01:27:40.719703 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.715904    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:27:40.721796 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.721764    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:40.795631 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.795596    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.795824 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.795645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.795824 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.795670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.795824 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.795718    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.795826    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-config\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.795867    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.795936    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfec3aee-387b-4b53-8e48-760e99709d2e-config-out\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.795965    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.795990    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796011    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796029    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-web-config\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796106    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796122    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67mvn\" (UniqueName: \"kubernetes.io/projected/bfec3aee-387b-4b53-8e48-760e99709d2e-kube-api-access-67mvn\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796167    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796187    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796225    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796249    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796716 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796268    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfec3aee-387b-4b53-8e48-760e99709d2e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.796716 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.796287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.898608 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898550    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.898767 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898638    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.898767 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.898767 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898687    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.898890 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898817    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.898890 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-config\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.898890 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898874    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899037 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898911    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfec3aee-387b-4b53-8e48-760e99709d2e-config-out\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899037 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898936    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899037 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.898966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899037 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899003    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899037 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899029    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899060    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-web-config\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899100    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899169    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67mvn\" (UniqueName: \"kubernetes.io/projected/bfec3aee-387b-4b53-8e48-760e99709d2e-kube-api-access-67mvn\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899243    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899255    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.899280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899273    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.900161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899318    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.900161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899349    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfec3aee-387b-4b53-8e48-760e99709d2e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.900161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899453    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.900161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899587    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.900161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.899740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.900161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.900105    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.900161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.900105    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.904264 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.904204    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.904377 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.904293    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bfec3aee-387b-4b53-8e48-760e99709d2e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.904687 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.904661    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.904776 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.904730    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.905008 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.904982    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-config\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.905246 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.905197    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bfec3aee-387b-4b53-8e48-760e99709d2e-config-out\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.905352 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.905285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.905415 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.905382    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-web-config\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.905800 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.905776    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.905942 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.905901    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.906102 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.906083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bfec3aee-387b-4b53-8e48-760e99709d2e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.908133 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.906873    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.908133 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.907226    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bfec3aee-387b-4b53-8e48-760e99709d2e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.914496 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.914476    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67mvn\" (UniqueName: \"kubernetes.io/projected/bfec3aee-387b-4b53-8e48-760e99709d2e-kube-api-access-67mvn\") pod \"prometheus-k8s-0\" (UID: \"bfec3aee-387b-4b53-8e48-760e99709d2e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:40.989071 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.988962    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg" event={"ID":"e1ce6711-b7fb-4482-a04d-3b4747e45c99","Type":"ContainerStarted","Data":"3d39260f428b1794a4331805cda1043492fb3d1cecbf6d1e250ab3d94556896c"}
Jul 28 01:27:40.989722 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.989241    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg" event={"ID":"e1ce6711-b7fb-4482-a04d-3b4747e45c99","Type":"ContainerStarted","Data":"6ceb9fa5e507eb847ab386a60320441360df9cda046449822e50bb226ce2507d"}
Jul 28 01:27:40.989722 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:40.989287    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg" event={"ID":"e1ce6711-b7fb-4482-a04d-3b4747e45c99","Type":"ContainerStarted","Data":"fa8b04c838034c819f4efdc5f1a0849f96f6e427f909f8cecbb675535a5aa032"}
Jul 28 01:27:41.011177 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:41.011075    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-95c7f4b44-2t84p" podStartSLOduration=2.011053817 podStartE2EDuration="2.011053817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:41.008850152 +0000 UTC m=+141.181595402" watchObservedRunningTime="2026-07-28 01:27:41.011053817 +0000 UTC m=+141.183799067"
Jul 28 01:27:41.021002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:41.020975    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:41.307042 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:41.307019    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:27:41.410312 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:27:41.410283    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfec3aee_387b_4b53_8e48_760e99709d2e.slice/crio-06c2ad13053db9e1c006d32dcd03df46d72281112307b93a28584ed6356994f3 WatchSource:0}: Error finding container 06c2ad13053db9e1c006d32dcd03df46d72281112307b93a28584ed6356994f3: Status 404 returned error can't find the container with id 06c2ad13053db9e1c006d32dcd03df46d72281112307b93a28584ed6356994f3
Jul 28 01:27:41.995801 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:41.995761    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg" event={"ID":"e1ce6711-b7fb-4482-a04d-3b4747e45c99","Type":"ContainerStarted","Data":"47a97b6b1509e687b00e885fb583c246dd69e4191177c71dd7afc95c9c1ff53e"}
Jul 28 01:27:41.996284 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:41.995840    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg" event={"ID":"e1ce6711-b7fb-4482-a04d-3b4747e45c99","Type":"ContainerStarted","Data":"aa8ca805643ffbb5c0dd23dfa086e5d3dce93e8a0eea982a98e4e4e198a50ebf"}
Jul 28 01:27:41.996284 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:41.995880    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:41.996284 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:41.995914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg" event={"ID":"e1ce6711-b7fb-4482-a04d-3b4747e45c99","Type":"ContainerStarted","Data":"bbc08aeed583f8d9f4168098c66f525a4befe8ed8dcb0b4ae2784fc9102f694e"}
Jul 28 01:27:41.997004 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:41.996982    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfec3aee-387b-4b53-8e48-760e99709d2e","Type":"ContainerStarted","Data":"06c2ad13053db9e1c006d32dcd03df46d72281112307b93a28584ed6356994f3"}
Jul 28 01:27:42.024643 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:42.024597    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg" podStartSLOduration=1.841119371 podStartE2EDuration="5.024581884s" totalImagesPullingTime="3.183462513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:42.022180676 +0000 UTC m=+142.194925961" watchObservedRunningTime="2026-07-28 01:27:42.024581884 +0000 UTC m=+142.197327133"
Jul 28 01:27:42.141065 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:42.141033    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:42.242918 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:42.242859    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:43.002797 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:43.002720    2556 generic.go:356] "Generic (PLEG): container finished" podID="bfec3aee-387b-4b53-8e48-760e99709d2e" containerID="b5115154e6289e7abf59e67a48c24dc1fd3605a7848d5114ea684cb34f16a890" exitCode=0
Jul 28 01:27:43.003261 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:43.002801    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfec3aee-387b-4b53-8e48-760e99709d2e","Type":"ContainerDied","Data":"b5115154e6289e7abf59e67a48c24dc1fd3605a7848d5114ea684cb34f16a890"}
Jul 28 01:27:44.622228 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:44.622190    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:44.741430 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:44.741399    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:44.817059 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:44.817023    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:47.026368 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:47.026342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfec3aee-387b-4b53-8e48-760e99709d2e","Type":"ContainerStarted","Data":"28d91b4be6de6ea5d911ccda5f6b8ad249772fd95ec849e33cd034205559677d"}
Jul 28 01:27:47.026682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:47.026438    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfec3aee-387b-4b53-8e48-760e99709d2e","Type":"ContainerStarted","Data":"409de7c90dad45936f568b7a0858c0ba15347684f239d2231e3ed544de31c5cd"}
Jul 28 01:27:47.026682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:47.026467    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfec3aee-387b-4b53-8e48-760e99709d2e","Type":"ContainerStarted","Data":"ff2840774fb23757c630a62bb806160509bc50f1770b0f2e3c21dfa911d4c77f"}
Jul 28 01:27:47.026682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:47.026497    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfec3aee-387b-4b53-8e48-760e99709d2e","Type":"ContainerStarted","Data":"cc20e0429d4aa5de9fe0b87df92079ef53276eb2c133619da15f6536b230cccf"}
Jul 28 01:27:48.009355 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:48.009325    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5988bb7987-nsstg"
Jul 28 01:27:48.033844 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:48.033815    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfec3aee-387b-4b53-8e48-760e99709d2e","Type":"ContainerStarted","Data":"a991e001b936c572da075018bb2cb903a0cc875b398c45a9df3873e790787d6f"}
Jul 28 01:27:48.034204 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:48.033872    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bfec3aee-387b-4b53-8e48-760e99709d2e","Type":"ContainerStarted","Data":"b7fc5ea6d5ba943602bc5980a90a7d6d5ac3d66a3f3bce94ef63f2533edab5a4"}
Jul 28 01:27:48.063140 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:48.063075    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.286867681 podStartE2EDuration="8.063055503s" totalImagesPullingTime="4.746802493s" totalInitContainerRuntime="29.385329ms" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:48.061304309 +0000 UTC m=+148.234049560" watchObservedRunningTime="2026-07-28 01:27:48.063055503 +0000 UTC m=+148.235800754"
Jul 28 01:27:49.474348 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:49.474314    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:49.474842 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:49.474383    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:49.478629 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:49.478609    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:50.045647 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:50.045619    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:27:50.101037 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:50.100993    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54b6c76676-m6kqc"]
Jul 28 01:27:50.266785 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:50.266755    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:50.364695 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:50.364661    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:50.420447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:50.420413    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:51.021478 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:51.021445    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:53.210618 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:53.210574    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:53.273886 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:53.273835    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:58.224664 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.224623    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7" podUID="b2710cee-558f-444f-82a2-286ce35dbf62" containerName="registry" containerID="cri-o://71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5" gracePeriod=30
Jul 28 01:27:58.508119 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.508065    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:27:58.670010 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.669971    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-installation-pull-secrets\") pod \"b2710cee-558f-444f-82a2-286ce35dbf62\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") "
Jul 28 01:27:58.670198 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.670046    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhfw6\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-kube-api-access-bhfw6\") pod \"b2710cee-558f-444f-82a2-286ce35dbf62\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") "
Jul 28 01:27:58.670198 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.670063    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-bound-sa-token\") pod \"b2710cee-558f-444f-82a2-286ce35dbf62\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") "
Jul 28 01:27:58.670315 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.670236    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-trusted-ca\") pod \"b2710cee-558f-444f-82a2-286ce35dbf62\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") "
Jul 28 01:27:58.670373 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.670356    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-registry-certificates\") pod \"b2710cee-558f-444f-82a2-286ce35dbf62\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") "
Jul 28 01:27:58.670434 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.670396    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2710cee-558f-444f-82a2-286ce35dbf62-ca-trust-extracted\") pod \"b2710cee-558f-444f-82a2-286ce35dbf62\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") "
Jul 28 01:27:58.670434 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.670429    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") pod \"b2710cee-558f-444f-82a2-286ce35dbf62\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") "
Jul 28 01:27:58.670533 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.670482    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-image-registry-private-configuration\") pod \"b2710cee-558f-444f-82a2-286ce35dbf62\" (UID: \"b2710cee-558f-444f-82a2-286ce35dbf62\") "
Jul 28 01:27:58.670801 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.670772    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b2710cee-558f-444f-82a2-286ce35dbf62" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:58.670913 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.670812    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b2710cee-558f-444f-82a2-286ce35dbf62" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:58.673072 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.673040    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b2710cee-558f-444f-82a2-286ce35dbf62" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:58.673173 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.673144    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b2710cee-558f-444f-82a2-286ce35dbf62" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:58.673235 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.673203    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b2710cee-558f-444f-82a2-286ce35dbf62" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:58.673276 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.673228    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-kube-api-access-bhfw6" (OuterVolumeSpecName: "kube-api-access-bhfw6") pod "b2710cee-558f-444f-82a2-286ce35dbf62" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62"). InnerVolumeSpecName "kube-api-access-bhfw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:58.673385 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.673366    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b2710cee-558f-444f-82a2-286ce35dbf62" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:58.679478 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.679457    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2710cee-558f-444f-82a2-286ce35dbf62-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b2710cee-558f-444f-82a2-286ce35dbf62" (UID: "b2710cee-558f-444f-82a2-286ce35dbf62"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:27:58.771902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.771840    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhfw6\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-kube-api-access-bhfw6\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:58.771902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.771870    2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-bound-sa-token\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:58.771902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.771881    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-trusted-ca\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:58.771902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.771888    2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2710cee-558f-444f-82a2-286ce35dbf62-registry-certificates\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:58.771902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.771896    2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2710cee-558f-444f-82a2-286ce35dbf62-ca-trust-extracted\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:58.771902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.771904    2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2710cee-558f-444f-82a2-286ce35dbf62-registry-tls\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:58.772144 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.771913    2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-image-registry-private-configuration\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:58.772144 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:58.771922    2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2710cee-558f-444f-82a2-286ce35dbf62-installation-pull-secrets\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:59.014197 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.014156    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bcc8879b6-8sscg" podUID="c19807cf-e7da-493b-b41c-8f1f3d74610c" containerName="console" containerID="cri-o://efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a" gracePeriod=15
Jul 28 01:27:59.080101 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.080069    2556 generic.go:356] "Generic (PLEG): container finished" podID="b2710cee-558f-444f-82a2-286ce35dbf62" containerID="71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5" exitCode=0
Jul 28 01:27:59.080235 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.080136    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7"
Jul 28 01:27:59.080235 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.080147    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7" event={"ID":"b2710cee-558f-444f-82a2-286ce35dbf62","Type":"ContainerDied","Data":"71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5"}
Jul 28 01:27:59.080235 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.080213    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b64567b4-c7jm7" event={"ID":"b2710cee-558f-444f-82a2-286ce35dbf62","Type":"ContainerDied","Data":"37dfd2b7f2a07ab0caf35d27b285045b2dd50041cd677b45b824080c00d4fe1d"}
Jul 28 01:27:59.080235 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.080232    2556 scope.go:117] "RemoveContainer" containerID="71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5"
Jul 28 01:27:59.123327 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.123302    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69b64567b4-c7jm7"]
Jul 28 01:27:59.126779 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.126756    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69b64567b4-c7jm7"]
Jul 28 01:27:59.136935 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.136914    2556 scope.go:117] "RemoveContainer" containerID="71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5"
Jul 28 01:27:59.137238 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:27:59.137215    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5\": container with ID starting with 71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5 not found: ID does not exist" containerID="71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5"
Jul 28 01:27:59.137307 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.137248    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5"} err="failed to get container status \"71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5\": rpc error: code = NotFound desc = could not find container \"71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5\": container with ID starting with 71ec2e33b8d0f3c79f90170c353b0736b8f78de8af3b36cfac5f92bf81e0b1a5 not found: ID does not exist"
Jul 28 01:27:59.323845 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.323816    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:27:59.478082 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.478055    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-serving-cert\") pod \"c19807cf-e7da-493b-b41c-8f1f3d74610c\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") "
Jul 28 01:27:59.478253 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.478114    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-service-ca\") pod \"c19807cf-e7da-493b-b41c-8f1f3d74610c\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") "
Jul 28 01:27:59.478253 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.478139    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqr4r\" (UniqueName: \"kubernetes.io/projected/c19807cf-e7da-493b-b41c-8f1f3d74610c-kube-api-access-pqr4r\") pod \"c19807cf-e7da-493b-b41c-8f1f3d74610c\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") "
Jul 28 01:27:59.478253 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.478208    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-config\") pod \"c19807cf-e7da-493b-b41c-8f1f3d74610c\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") "
Jul 28 01:27:59.478253 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.478228    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-oauth-serving-cert\") pod \"c19807cf-e7da-493b-b41c-8f1f3d74610c\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") "
Jul 28 01:27:59.478468 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.478256    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-oauth-config\") pod \"c19807cf-e7da-493b-b41c-8f1f3d74610c\" (UID: \"c19807cf-e7da-493b-b41c-8f1f3d74610c\") "
Jul 28 01:27:59.478591 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.478527    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-config" (OuterVolumeSpecName: "console-config") pod "c19807cf-e7da-493b-b41c-8f1f3d74610c" (UID: "c19807cf-e7da-493b-b41c-8f1f3d74610c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:59.478591 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.478553    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-service-ca" (OuterVolumeSpecName: "service-ca") pod "c19807cf-e7da-493b-b41c-8f1f3d74610c" (UID: "c19807cf-e7da-493b-b41c-8f1f3d74610c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:59.478697 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.478648    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c19807cf-e7da-493b-b41c-8f1f3d74610c" (UID: "c19807cf-e7da-493b-b41c-8f1f3d74610c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:59.480517 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.480489    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c19807cf-e7da-493b-b41c-8f1f3d74610c" (UID: "c19807cf-e7da-493b-b41c-8f1f3d74610c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:59.480517 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.480493    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19807cf-e7da-493b-b41c-8f1f3d74610c-kube-api-access-pqr4r" (OuterVolumeSpecName: "kube-api-access-pqr4r") pod "c19807cf-e7da-493b-b41c-8f1f3d74610c" (UID: "c19807cf-e7da-493b-b41c-8f1f3d74610c"). InnerVolumeSpecName "kube-api-access-pqr4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:59.480728 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.480519    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c19807cf-e7da-493b-b41c-8f1f3d74610c" (UID: "c19807cf-e7da-493b-b41c-8f1f3d74610c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:59.579420 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.579357    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:59.579420 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.579380    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-oauth-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:59.579420 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.579389    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-oauth-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:59.579420 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.579398    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c19807cf-e7da-493b-b41c-8f1f3d74610c-console-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:59.579420 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.579407    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c19807cf-e7da-493b-b41c-8f1f3d74610c-service-ca\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:27:59.579420 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:27:59.579415    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqr4r\" (UniqueName: \"kubernetes.io/projected/c19807cf-e7da-493b-b41c-8f1f3d74610c-kube-api-access-pqr4r\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:28:00.086251 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.086219    2556 generic.go:356] "Generic (PLEG): container finished" podID="c19807cf-e7da-493b-b41c-8f1f3d74610c" containerID="efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a" exitCode=0
Jul 28 01:28:00.086425 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.086268    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bcc8879b6-8sscg"
Jul 28 01:28:00.086425 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.086279    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcc8879b6-8sscg" event={"ID":"c19807cf-e7da-493b-b41c-8f1f3d74610c","Type":"ContainerDied","Data":"efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a"}
Jul 28 01:28:00.086425 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.086320    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bcc8879b6-8sscg" event={"ID":"c19807cf-e7da-493b-b41c-8f1f3d74610c","Type":"ContainerDied","Data":"c382f35f1d672d246c7297549189f2a1d71a6e2a360830b8949a81824ece49ba"}
Jul 28 01:28:00.086425 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.086340    2556 scope.go:117] "RemoveContainer" containerID="efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a"
Jul 28 01:28:00.102179 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.102162    2556 scope.go:117] "RemoveContainer" containerID="efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a"
Jul 28 01:28:00.102430 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:28:00.102410    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a\": container with ID starting with efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a not found: ID does not exist" containerID="efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a"
Jul 28 01:28:00.102474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.102437    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a"} err="failed to get container status \"efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a\": rpc error: code = NotFound desc = could not find container \"efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a\": container with ID starting with efde7c34d365763fb2f560ddcba373f4366666ed779191913b8a2473bd468e8a not found: ID does not exist"
Jul 28 01:28:00.107729 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.107706    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bcc8879b6-8sscg"]
Jul 28 01:28:00.110668 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.110648    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bcc8879b6-8sscg"]
Jul 28 01:28:00.286349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.286304    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2710cee-558f-444f-82a2-286ce35dbf62" path="/var/lib/kubelet/pods/b2710cee-558f-444f-82a2-286ce35dbf62/volumes"
Jul 28 01:28:00.286761 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:00.286741    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c19807cf-e7da-493b-b41c-8f1f3d74610c" path="/var/lib/kubelet/pods/c19807cf-e7da-493b-b41c-8f1f3d74610c/volumes"
Jul 28 01:28:01.496126 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:01.496095    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:01.567261 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:01.567233    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:08.212996 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:08.212954    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:08.314488 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:08.314458    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:15.121014 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.120909    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54b6c76676-m6kqc" podUID="d9fb96d4-f63d-43b1-89de-39597f1f5235" containerName="console" containerID="cri-o://84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7" gracePeriod=15
Jul 28 01:28:15.387983 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.387917    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:28:15.492789 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.492759    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-service-ca\") pod \"d9fb96d4-f63d-43b1-89de-39597f1f5235\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") "
Jul 28 01:28:15.492987 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.492817    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-trusted-ca-bundle\") pod \"d9fb96d4-f63d-43b1-89de-39597f1f5235\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") "
Jul 28 01:28:15.492987 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.492837    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-serving-cert\") pod \"d9fb96d4-f63d-43b1-89de-39597f1f5235\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") "
Jul 28 01:28:15.492987 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.492873    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlwtq\" (UniqueName: \"kubernetes.io/projected/d9fb96d4-f63d-43b1-89de-39597f1f5235-kube-api-access-vlwtq\") pod \"d9fb96d4-f63d-43b1-89de-39597f1f5235\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") "
Jul 28 01:28:15.493160 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.493083    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-config\") pod \"d9fb96d4-f63d-43b1-89de-39597f1f5235\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") "
Jul 28 01:28:15.493160 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.493132    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-oauth-config\") pod \"d9fb96d4-f63d-43b1-89de-39597f1f5235\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") "
Jul 28 01:28:15.493263 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.493193    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-oauth-serving-cert\") pod \"d9fb96d4-f63d-43b1-89de-39597f1f5235\" (UID: \"d9fb96d4-f63d-43b1-89de-39597f1f5235\") "
Jul 28 01:28:15.493263 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.493203    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-service-ca" (OuterVolumeSpecName: "service-ca") pod "d9fb96d4-f63d-43b1-89de-39597f1f5235" (UID: "d9fb96d4-f63d-43b1-89de-39597f1f5235"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:15.493381 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.493348    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d9fb96d4-f63d-43b1-89de-39597f1f5235" (UID: "d9fb96d4-f63d-43b1-89de-39597f1f5235"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:15.493440 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.493415    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-config" (OuterVolumeSpecName: "console-config") pod "d9fb96d4-f63d-43b1-89de-39597f1f5235" (UID: "d9fb96d4-f63d-43b1-89de-39597f1f5235"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:15.493489 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.493474    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-trusted-ca-bundle\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:28:15.493533 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.493494    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-service-ca\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:28:15.493626 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.493611    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d9fb96d4-f63d-43b1-89de-39597f1f5235" (UID: "d9fb96d4-f63d-43b1-89de-39597f1f5235"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:15.495343 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.495319    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d9fb96d4-f63d-43b1-89de-39597f1f5235" (UID: "d9fb96d4-f63d-43b1-89de-39597f1f5235"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:15.495609 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.495592    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d9fb96d4-f63d-43b1-89de-39597f1f5235" (UID: "d9fb96d4-f63d-43b1-89de-39597f1f5235"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:15.495702 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.495681    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9fb96d4-f63d-43b1-89de-39597f1f5235-kube-api-access-vlwtq" (OuterVolumeSpecName: "kube-api-access-vlwtq") pod "d9fb96d4-f63d-43b1-89de-39597f1f5235" (UID: "d9fb96d4-f63d-43b1-89de-39597f1f5235"). InnerVolumeSpecName "kube-api-access-vlwtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:28:15.594776 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.594743    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-oauth-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:28:15.594776 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.594771    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:28:15.594776 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.594781    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlwtq\" (UniqueName: \"kubernetes.io/projected/d9fb96d4-f63d-43b1-89de-39597f1f5235-kube-api-access-vlwtq\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:28:15.594982 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.594791    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:28:15.594982 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:15.594799    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9fb96d4-f63d-43b1-89de-39597f1f5235-console-oauth-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:28:16.150291 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.150257    2556 generic.go:356] "Generic (PLEG): container finished" podID="d9fb96d4-f63d-43b1-89de-39597f1f5235" containerID="84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7" exitCode=0
Jul 28 01:28:16.150752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.150305    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b6c76676-m6kqc"
Jul 28 01:28:16.150752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.150343    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b6c76676-m6kqc" event={"ID":"d9fb96d4-f63d-43b1-89de-39597f1f5235","Type":"ContainerDied","Data":"84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7"}
Jul 28 01:28:16.150752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.150395    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b6c76676-m6kqc" event={"ID":"d9fb96d4-f63d-43b1-89de-39597f1f5235","Type":"ContainerDied","Data":"862111c4f58435b9588b56c275ae65d10517b56592fe55b67c1c104d48bf4dfd"}
Jul 28 01:28:16.150752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.150416    2556 scope.go:117] "RemoveContainer" containerID="84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7"
Jul 28 01:28:16.159124 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.159108    2556 scope.go:117] "RemoveContainer" containerID="84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7"
Jul 28 01:28:16.159362 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:28:16.159345    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7\": container with ID starting with 84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7 not found: ID does not exist" containerID="84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7"
Jul 28 01:28:16.159411 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.159371    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7"} err="failed to get container status \"84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7\": rpc error: code = NotFound desc = could not find container \"84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7\": container with ID starting with 84f634af9fd8f69c2910afc705fc2cd4c562ee23c5b60934bae8060ebcf90ce7 not found: ID does not exist"
Jul 28 01:28:16.172395 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.172375    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54b6c76676-m6kqc"]
Jul 28 01:28:16.178126 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.178107    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54b6c76676-m6kqc"]
Jul 28 01:28:16.285880 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:16.285856    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9fb96d4-f63d-43b1-89de-39597f1f5235" path="/var/lib/kubelet/pods/d9fb96d4-f63d-43b1-89de-39597f1f5235/volumes"
Jul 28 01:28:28.503132 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:28.503077    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:41.021816 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:41.021775    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:41.042374 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:41.042349    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:41.263445 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:41.263417    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:53.692294 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:53.692259    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:53.789202 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:28:53.789163    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:00.333721 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:00.333677    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:00.435402 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:00.435365    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:02.354121 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354087    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-658bf6b5f8-nkgtz"]
Jul 28 01:29:02.354488 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354438    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2710cee-558f-444f-82a2-286ce35dbf62" containerName="registry"
Jul 28 01:29:02.354488 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354450    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2710cee-558f-444f-82a2-286ce35dbf62" containerName="registry"
Jul 28 01:29:02.354488 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354474    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9fb96d4-f63d-43b1-89de-39597f1f5235" containerName="console"
Jul 28 01:29:02.354488 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354479    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9fb96d4-f63d-43b1-89de-39597f1f5235" containerName="console"
Jul 28 01:29:02.354488 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354489    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19807cf-e7da-493b-b41c-8f1f3d74610c" containerName="console"
Jul 28 01:29:02.354688 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354494    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19807cf-e7da-493b-b41c-8f1f3d74610c" containerName="console"
Jul 28 01:29:02.354688 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354542    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9fb96d4-f63d-43b1-89de-39597f1f5235" containerName="console"
Jul 28 01:29:02.354688 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354568    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2710cee-558f-444f-82a2-286ce35dbf62" containerName="registry"
Jul 28 01:29:02.354688 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.354574    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19807cf-e7da-493b-b41c-8f1f3d74610c" containerName="console"
Jul 28 01:29:02.357170 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.357153    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.368590 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.368544    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-658bf6b5f8-nkgtz"]
Jul 28 01:29:02.417964 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.417922    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:02.479034 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.479006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-config\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.479364 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.479060    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-oauth-serving-cert\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.479364 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.479143    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-service-ca\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.479364 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.479168    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-trusted-ca-bundle\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.479364 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.479196    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-oauth-config\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.479364 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.479231    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-serving-cert\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.479364 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.479248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vdc\" (UniqueName: \"kubernetes.io/projected/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-kube-api-access-64vdc\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.480823 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.480792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:02.580646 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.580614    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-config\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.580823 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.580661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-oauth-serving-cert\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.580823 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.580791    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-service-ca\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.580823 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.580823    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-trusted-ca-bundle\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.580983 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.580855    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-oauth-config\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.580983 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.580885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-serving-cert\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.580983 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.580902    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vdc\" (UniqueName: \"kubernetes.io/projected/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-kube-api-access-64vdc\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.587082 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.581966    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-oauth-serving-cert\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.587082 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.582133    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-trusted-ca-bundle\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.587082 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.582189    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-service-ca\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.587082 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.582295    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-config\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.587747 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.587729    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-oauth-config\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.587802 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.587785    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-serving-cert\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.591782 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.591748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vdc\" (UniqueName: \"kubernetes.io/projected/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-kube-api-access-64vdc\") pod \"console-658bf6b5f8-nkgtz\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") " pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.666778 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.666694    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:02.795309 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:02.795278    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-658bf6b5f8-nkgtz"]
Jul 28 01:29:02.799160 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:29:02.799133    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3303ce94_ad9c_4c37_9f8a_e50ec9aba5f4.slice/crio-5a90e46c133b8c63b0be30da165cc48b1b217e5da950c1ccd60171eee2d17f51 WatchSource:0}: Error finding container 5a90e46c133b8c63b0be30da165cc48b1b217e5da950c1ccd60171eee2d17f51: Status 404 returned error can't find the container with id 5a90e46c133b8c63b0be30da165cc48b1b217e5da950c1ccd60171eee2d17f51
Jul 28 01:29:03.332101 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:03.332063    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-658bf6b5f8-nkgtz" event={"ID":"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4","Type":"ContainerStarted","Data":"a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f"}
Jul 28 01:29:03.332313 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:03.332123    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-658bf6b5f8-nkgtz" event={"ID":"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4","Type":"ContainerStarted","Data":"5a90e46c133b8c63b0be30da165cc48b1b217e5da950c1ccd60171eee2d17f51"}
Jul 28 01:29:03.349953 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:03.349904    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-658bf6b5f8-nkgtz" podStartSLOduration=1.349888886 podStartE2EDuration="1.349888886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:29:03.349368142 +0000 UTC m=+223.522113410" watchObservedRunningTime="2026-07-28 01:29:03.349888886 +0000 UTC m=+223.522634134"
Jul 28 01:29:04.437493 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:04.437455    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:08.912056 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:08.912021    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:08.976803 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:08.976775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:10.529486 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.529445    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-658bf6b5f8-nkgtz"]
Jul 28 01:29:10.564374 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.564348    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b8d7c6dc9-52jr2"]
Jul 28 01:29:10.566940 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.566922    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.576855 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.576833    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8d7c6dc9-52jr2"]
Jul 28 01:29:10.640873 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.640840    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-trusted-ca-bundle\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.640873 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.640873    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rbc\" (UniqueName: \"kubernetes.io/projected/1a954194-6a84-49d8-94e8-dd42a9cd8542-kube-api-access-87rbc\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.641061 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.640906    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-serving-cert\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.641061 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.640974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-oauth-config\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.641061 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.641006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-service-ca\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.641061 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.641042    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-oauth-serving-cert\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.641202 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.641075    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-config\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.742238 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.742203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-serving-cert\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.742238 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.742243    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-oauth-config\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.742437 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.742262    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-service-ca\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.742437 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.742291    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-oauth-serving-cert\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.742437 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.742326    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-config\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.742437 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.742348    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-trusted-ca-bundle\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.742437 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.742364    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rbc\" (UniqueName: \"kubernetes.io/projected/1a954194-6a84-49d8-94e8-dd42a9cd8542-kube-api-access-87rbc\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.743055 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.743035    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-service-ca\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.743304 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.743280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-trusted-ca-bundle\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.743379 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.743318    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-oauth-serving-cert\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.743415 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.743386    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-config\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.744911 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.744890    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-serving-cert\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.745059 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.745037    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-oauth-config\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.750513 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.750488    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rbc\" (UniqueName: \"kubernetes.io/projected/1a954194-6a84-49d8-94e8-dd42a9cd8542-kube-api-access-87rbc\") pod \"console-6b8d7c6dc9-52jr2\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") " pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:10.876836 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:10.876807    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:11.003701 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:11.003634    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8d7c6dc9-52jr2"]
Jul 28 01:29:11.006085 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:29:11.006043    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a954194_6a84_49d8_94e8_dd42a9cd8542.slice/crio-27a24f023835e3a1ded54bd82cf6ca1ca4ccada0c6931eecc026317d340bbd9e WatchSource:0}: Error finding container 27a24f023835e3a1ded54bd82cf6ca1ca4ccada0c6931eecc026317d340bbd9e: Status 404 returned error can't find the container with id 27a24f023835e3a1ded54bd82cf6ca1ca4ccada0c6931eecc026317d340bbd9e
Jul 28 01:29:11.363047 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:11.363015    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d7c6dc9-52jr2" event={"ID":"1a954194-6a84-49d8-94e8-dd42a9cd8542","Type":"ContainerStarted","Data":"b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b"}
Jul 28 01:29:11.363231 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:11.363072    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d7c6dc9-52jr2" event={"ID":"1a954194-6a84-49d8-94e8-dd42a9cd8542","Type":"ContainerStarted","Data":"27a24f023835e3a1ded54bd82cf6ca1ca4ccada0c6931eecc026317d340bbd9e"}
Jul 28 01:29:11.381247 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:11.381179    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b8d7c6dc9-52jr2" podStartSLOduration=1.381162624 podStartE2EDuration="1.381162624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:29:11.380079499 +0000 UTC m=+231.552824749" watchObservedRunningTime="2026-07-28 01:29:11.381162624 +0000 UTC m=+231.553907874"
Jul 28 01:29:12.667527 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:12.667488    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:20.877140 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:20.877109    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:20.877619 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:20.877171    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:20.881852 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:20.881830    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:20.951151 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:20.951116    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:21.008166 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:21.008131    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:21.405244 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:21.405213    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:29:21.450667 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:21.450637    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95c7f4b44-2t84p"]
Jul 28 01:29:21.551507 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:21.551468    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:21.728141 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:21.728059    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:21.784049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:21.784018    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:35.548646 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.548610    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-658bf6b5f8-nkgtz" podUID="3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" containerName="console" containerID="cri-o://a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f" gracePeriod=15
Jul 28 01:29:35.786210 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.786184    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:35.860165 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860134    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-serving-cert\") pod \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") "
Jul 28 01:29:35.860331 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860182    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-oauth-config\") pod \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") "
Jul 28 01:29:35.860331 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860224    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-service-ca\") pod \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") "
Jul 28 01:29:35.860331 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860254    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64vdc\" (UniqueName: \"kubernetes.io/projected/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-kube-api-access-64vdc\") pod \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") "
Jul 28 01:29:35.860331 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860286    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-oauth-serving-cert\") pod \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") "
Jul 28 01:29:35.860331 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860319    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-trusted-ca-bundle\") pod \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") "
Jul 28 01:29:35.860599 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860356    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-config\") pod \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\" (UID: \"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4\") "
Jul 28 01:29:35.860780 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860747    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-service-ca" (OuterVolumeSpecName: "service-ca") pod "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" (UID: "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:35.860905 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860749    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" (UID: "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:35.860905 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860861    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" (UID: "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:35.861013 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.860908    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-config" (OuterVolumeSpecName: "console-config") pod "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" (UID: "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:35.862506 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.862485    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-kube-api-access-64vdc" (OuterVolumeSpecName: "kube-api-access-64vdc") pod "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" (UID: "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4"). InnerVolumeSpecName "kube-api-access-64vdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:29:35.863062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.863044    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" (UID: "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:29:35.863115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.863063    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" (UID: "3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:29:35.961350 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.961324    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-trusted-ca-bundle\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:35.961350 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.961348    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:35.961546 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.961357    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:35.961546 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.961365    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-console-oauth-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:35.961546 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.961374    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-service-ca\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:35.961546 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.961383    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vdc\" (UniqueName: \"kubernetes.io/projected/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-kube-api-access-64vdc\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:35.961546 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:35.961392    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4-oauth-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:36.144964 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.144875    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:36.196042 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.196015    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:36.458211 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.458131    2556 generic.go:356] "Generic (PLEG): container finished" podID="3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" containerID="a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f" exitCode=0
Jul 28 01:29:36.458211 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.458181    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-658bf6b5f8-nkgtz"
Jul 28 01:29:36.458397 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.458182    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-658bf6b5f8-nkgtz" event={"ID":"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4","Type":"ContainerDied","Data":"a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f"}
Jul 28 01:29:36.458397 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.458309    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-658bf6b5f8-nkgtz" event={"ID":"3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4","Type":"ContainerDied","Data":"5a90e46c133b8c63b0be30da165cc48b1b217e5da950c1ccd60171eee2d17f51"}
Jul 28 01:29:36.458397 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.458337    2556 scope.go:117] "RemoveContainer" containerID="a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f"
Jul 28 01:29:36.467055 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.467027    2556 scope.go:117] "RemoveContainer" containerID="a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f"
Jul 28 01:29:36.472152 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:29:36.472125    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f\": container with ID starting with a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f not found: ID does not exist" containerID="a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f"
Jul 28 01:29:36.472254 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.472157    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f"} err="failed to get container status \"a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f\": rpc error: code = NotFound desc = could not find container \"a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f\": container with ID starting with a5499fca1e1d42a41c87311a169e14a6b8d0f386692b8222f4b81b4351128c0f not found: ID does not exist"
Jul 28 01:29:36.477867 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.477847    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-658bf6b5f8-nkgtz"]
Jul 28 01:29:36.481084 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:36.481066    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-658bf6b5f8-nkgtz"]
Jul 28 01:29:38.218812 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:38.218781    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:38.285821 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:38.285785    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" path="/var/lib/kubelet/pods/3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4/volumes"
Jul 28 01:29:38.316076 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:38.316044    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:46.470020 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.469964    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-95c7f4b44-2t84p" podUID="ddf8af51-d279-41e8-a9ed-97b371159291" containerName="console" containerID="cri-o://e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184" gracePeriod=15
Jul 28 01:29:46.718419 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.718398    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:29:46.854547 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.854513    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-service-ca\") pod \"ddf8af51-d279-41e8-a9ed-97b371159291\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") "
Jul 28 01:29:46.854760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.854627    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-console-config\") pod \"ddf8af51-d279-41e8-a9ed-97b371159291\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") "
Jul 28 01:29:46.854760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.854654    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-serving-cert\") pod \"ddf8af51-d279-41e8-a9ed-97b371159291\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") "
Jul 28 01:29:46.854760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.854676    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhndc\" (UniqueName: \"kubernetes.io/projected/ddf8af51-d279-41e8-a9ed-97b371159291-kube-api-access-zhndc\") pod \"ddf8af51-d279-41e8-a9ed-97b371159291\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") "
Jul 28 01:29:46.854760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.854708    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-trusted-ca-bundle\") pod \"ddf8af51-d279-41e8-a9ed-97b371159291\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") "
Jul 28 01:29:46.854760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.854738    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-oauth-config\") pod \"ddf8af51-d279-41e8-a9ed-97b371159291\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") "
Jul 28 01:29:46.855007 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.854762    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-oauth-serving-cert\") pod \"ddf8af51-d279-41e8-a9ed-97b371159291\" (UID: \"ddf8af51-d279-41e8-a9ed-97b371159291\") "
Jul 28 01:29:46.855062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.855022    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-service-ca" (OuterVolumeSpecName: "service-ca") pod "ddf8af51-d279-41e8-a9ed-97b371159291" (UID: "ddf8af51-d279-41e8-a9ed-97b371159291"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:46.855113 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.855060    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-console-config" (OuterVolumeSpecName: "console-config") pod "ddf8af51-d279-41e8-a9ed-97b371159291" (UID: "ddf8af51-d279-41e8-a9ed-97b371159291"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:46.855272 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.855240    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ddf8af51-d279-41e8-a9ed-97b371159291" (UID: "ddf8af51-d279-41e8-a9ed-97b371159291"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:46.855272 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.855252    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ddf8af51-d279-41e8-a9ed-97b371159291" (UID: "ddf8af51-d279-41e8-a9ed-97b371159291"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:46.857091 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.857058    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ddf8af51-d279-41e8-a9ed-97b371159291" (UID: "ddf8af51-d279-41e8-a9ed-97b371159291"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:29:46.857091 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.857073    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ddf8af51-d279-41e8-a9ed-97b371159291" (UID: "ddf8af51-d279-41e8-a9ed-97b371159291"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:29:46.857220 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.857148    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf8af51-d279-41e8-a9ed-97b371159291-kube-api-access-zhndc" (OuterVolumeSpecName: "kube-api-access-zhndc") pod "ddf8af51-d279-41e8-a9ed-97b371159291" (UID: "ddf8af51-d279-41e8-a9ed-97b371159291"). InnerVolumeSpecName "kube-api-access-zhndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:29:46.955818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.955788    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-service-ca\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:46.955818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.955814    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-console-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:46.955818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.955823    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:46.956027 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.955834    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhndc\" (UniqueName: \"kubernetes.io/projected/ddf8af51-d279-41e8-a9ed-97b371159291-kube-api-access-zhndc\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:46.956027 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.955862    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-trusted-ca-bundle\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:46.956027 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.955870    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddf8af51-d279-41e8-a9ed-97b371159291-console-oauth-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:46.956027 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:46.955878    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddf8af51-d279-41e8-a9ed-97b371159291-oauth-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:29:47.502430 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:47.502396    2556 generic.go:356] "Generic (PLEG): container finished" podID="ddf8af51-d279-41e8-a9ed-97b371159291" containerID="e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184" exitCode=0
Jul 28 01:29:47.502877 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:47.502439    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c7f4b44-2t84p"
Jul 28 01:29:47.502877 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:47.502458    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c7f4b44-2t84p" event={"ID":"ddf8af51-d279-41e8-a9ed-97b371159291","Type":"ContainerDied","Data":"e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184"}
Jul 28 01:29:47.502877 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:47.502501    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c7f4b44-2t84p" event={"ID":"ddf8af51-d279-41e8-a9ed-97b371159291","Type":"ContainerDied","Data":"512be241cb1e43751d7e5b0a92b76ade333b131d56f169979d32ec45633bbcab"}
Jul 28 01:29:47.502877 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:47.502523    2556 scope.go:117] "RemoveContainer" containerID="e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184"
Jul 28 01:29:47.511626 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:47.511607    2556 scope.go:117] "RemoveContainer" containerID="e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184"
Jul 28 01:29:47.511874 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:29:47.511856    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184\": container with ID starting with e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184 not found: ID does not exist" containerID="e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184"
Jul 28 01:29:47.511926 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:47.511882    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184"} err="failed to get container status \"e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184\": rpc error: code = NotFound desc = could not find container \"e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184\": container with ID starting with e36738c4830c126b3e744dfe61a7c165e2a474df6a963e5ac818c4e0d40e5184 not found: ID does not exist"
Jul 28 01:29:47.524520 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:47.524497    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95c7f4b44-2t84p"]
Jul 28 01:29:47.526361 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:47.526342    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-95c7f4b44-2t84p"]
Jul 28 01:29:48.285859 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:29:48.285825    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf8af51-d279-41e8-a9ed-97b371159291" path="/var/lib/kubelet/pods/ddf8af51-d279-41e8-a9ed-97b371159291/volumes"
Jul 28 01:30:20.222298 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:20.222263    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:30:20.228747 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:20.228724    2556 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 01:30:20.229016 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:20.228999    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:30:32.830359 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:32.830319    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:32.884687 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:32.884657    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:34.064548 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:34.064517    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:34.162345 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:34.162314    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:34.438731 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:34.438697    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:34.539323 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:34.539290    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:34.598979 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:34.598944    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:57.171700 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:57.171665    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:57.263171 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:30:57.263134    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:02.874417 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:02.874375    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:04.191614 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:04.191578    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:04.276172 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:04.276140    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:04.395064 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:04.395032    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:04.481906 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:04.481824    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:34.749756 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:34.749685    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:34.819231 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:34.819202    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:55.030368 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:55.030315    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:56.956318 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:56.956283    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:57.058061 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:31:57.058032    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:21.387110 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:32:21.387074    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:21.498158 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:32:21.498124    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:21.585574 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:32:21.585530    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:21.861778 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:32:21.861741    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:22.062437 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:32:22.062404    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:22.178015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:32:22.177937    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:05.435230 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:05.435194    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:06.359817 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:06.359779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:16.689405 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:16.689371    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:17.442469 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:17.442435    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:20.275934 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:20.275908    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:20.376765 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:20.376730    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:22.639517 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.639483    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:22.805760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.805722    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:22.852732 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.852698    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5zrnd"]
Jul 28 01:33:22.853002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.852991    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf8af51-d279-41e8-a9ed-97b371159291" containerName="console"
Jul 28 01:33:22.853049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.853004    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf8af51-d279-41e8-a9ed-97b371159291" containerName="console"
Jul 28 01:33:22.853049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.853024    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" containerName="console"
Jul 28 01:33:22.853049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.853030    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" containerName="console"
Jul 28 01:33:22.853138 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.853080    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3303ce94-ad9c-4c37-9f8a-e50ec9aba5f4" containerName="console"
Jul 28 01:33:22.853138 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.853088    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf8af51-d279-41e8-a9ed-97b371159291" containerName="console"
Jul 28 01:33:22.855183 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.855170    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:22.858178 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.858159    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 01:33:22.858178 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.858173    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 01:33:22.858316 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.858174    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dkffb\""
Jul 28 01:33:22.865475 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.865455    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5zrnd"]
Jul 28 01:33:22.914146 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.914087    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mpr6\" (UniqueName: \"kubernetes.io/projected/b3fb3ff0-d871-479b-a8da-a09f61008f30-kube-api-access-6mpr6\") pod \"cert-manager-webhook-6486ffd997-5zrnd\" (UID: \"b3fb3ff0-d871-479b-a8da-a09f61008f30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:22.914248 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:22.914155    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3fb3ff0-d871-479b-a8da-a09f61008f30-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5zrnd\" (UID: \"b3fb3ff0-d871-479b-a8da-a09f61008f30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:23.014954 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:23.014926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3fb3ff0-d871-479b-a8da-a09f61008f30-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5zrnd\" (UID: \"b3fb3ff0-d871-479b-a8da-a09f61008f30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:23.015083 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:23.014966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mpr6\" (UniqueName: \"kubernetes.io/projected/b3fb3ff0-d871-479b-a8da-a09f61008f30-kube-api-access-6mpr6\") pod \"cert-manager-webhook-6486ffd997-5zrnd\" (UID: \"b3fb3ff0-d871-479b-a8da-a09f61008f30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:23.034085 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:23.034064    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3fb3ff0-d871-479b-a8da-a09f61008f30-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5zrnd\" (UID: \"b3fb3ff0-d871-479b-a8da-a09f61008f30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:23.034611 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:23.034590    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mpr6\" (UniqueName: \"kubernetes.io/projected/b3fb3ff0-d871-479b-a8da-a09f61008f30-kube-api-access-6mpr6\") pod \"cert-manager-webhook-6486ffd997-5zrnd\" (UID: \"b3fb3ff0-d871-479b-a8da-a09f61008f30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:23.178479 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:23.178406    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:23.308518 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:23.308494    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5zrnd"]
Jul 28 01:33:23.311372 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:33:23.311330    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3fb3ff0_d871_479b_a8da_a09f61008f30.slice/crio-88c14143369f36866843cf654541524aa20cc8807dbe9c550cbd22af90ddf7b6 WatchSource:0}: Error finding container 88c14143369f36866843cf654541524aa20cc8807dbe9c550cbd22af90ddf7b6: Status 404 returned error can't find the container with id 88c14143369f36866843cf654541524aa20cc8807dbe9c550cbd22af90ddf7b6
Jul 28 01:33:23.313235 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:23.313220    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:33:24.318848 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:24.318808    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd" event={"ID":"b3fb3ff0-d871-479b-a8da-a09f61008f30","Type":"ContainerStarted","Data":"88c14143369f36866843cf654541524aa20cc8807dbe9c550cbd22af90ddf7b6"}
Jul 28 01:33:24.386142 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:24.386103    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:25.045715 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.045682    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:25.212910 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.212866    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:25.281035 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.280995    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:25.486547 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.486513    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"]
Jul 28 01:33:25.495233 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.495178    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"
Jul 28 01:33:25.499366 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.499333    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gxs8t\""
Jul 28 01:33:25.505431 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.505381    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"]
Jul 28 01:33:25.538485 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.538457    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91d97570-bf8b-4072-a857-a0460b3e8919-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zlv7s\" (UID: \"91d97570-bf8b-4072-a857-a0460b3e8919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"
Jul 28 01:33:25.538645 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.538492    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjj4q\" (UniqueName: \"kubernetes.io/projected/91d97570-bf8b-4072-a857-a0460b3e8919-kube-api-access-qjj4q\") pod \"cert-manager-cainjector-559cdd8c87-zlv7s\" (UID: \"91d97570-bf8b-4072-a857-a0460b3e8919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"
Jul 28 01:33:25.639976 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.639936    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91d97570-bf8b-4072-a857-a0460b3e8919-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zlv7s\" (UID: \"91d97570-bf8b-4072-a857-a0460b3e8919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"
Jul 28 01:33:25.639976 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.639982    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjj4q\" (UniqueName: \"kubernetes.io/projected/91d97570-bf8b-4072-a857-a0460b3e8919-kube-api-access-qjj4q\") pod \"cert-manager-cainjector-559cdd8c87-zlv7s\" (UID: \"91d97570-bf8b-4072-a857-a0460b3e8919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"
Jul 28 01:33:25.660261 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.660219    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91d97570-bf8b-4072-a857-a0460b3e8919-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zlv7s\" (UID: \"91d97570-bf8b-4072-a857-a0460b3e8919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"
Jul 28 01:33:25.660461 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.660442    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjj4q\" (UniqueName: \"kubernetes.io/projected/91d97570-bf8b-4072-a857-a0460b3e8919-kube-api-access-qjj4q\") pod \"cert-manager-cainjector-559cdd8c87-zlv7s\" (UID: \"91d97570-bf8b-4072-a857-a0460b3e8919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"
Jul 28 01:33:25.808494 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:25.808412    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"
Jul 28 01:33:26.687286 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:26.687257    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s"]
Jul 28 01:33:26.689005 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:33:26.688961    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91d97570_bf8b_4072_a857_a0460b3e8919.slice/crio-e26b7b09dc80e927850b2d878dabc57f68e6e5c3bb26f6454deca4e43be13fd0 WatchSource:0}: Error finding container e26b7b09dc80e927850b2d878dabc57f68e6e5c3bb26f6454deca4e43be13fd0: Status 404 returned error can't find the container with id e26b7b09dc80e927850b2d878dabc57f68e6e5c3bb26f6454deca4e43be13fd0
Jul 28 01:33:26.960043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:26.959987    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:27.078533 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:27.078500    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:27.327914 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:27.327881    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:27.334902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:27.334871    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd" event={"ID":"b3fb3ff0-d871-479b-a8da-a09f61008f30","Type":"ContainerStarted","Data":"b53538c7b85da70214f6ef2739e5827d9d1f10b80f0bfca04485f393ff63db1f"}
Jul 28 01:33:27.335051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:27.335033    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:27.336768 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:27.336747    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s" event={"ID":"91d97570-bf8b-4072-a857-a0460b3e8919","Type":"ContainerStarted","Data":"622c29cbd098dc76cd3c91545a63a50d9d4c41129d4d5194deb7dfbfacb468c0"}
Jul 28 01:33:27.336870 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:27.336776    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s" event={"ID":"91d97570-bf8b-4072-a857-a0460b3e8919","Type":"ContainerStarted","Data":"e26b7b09dc80e927850b2d878dabc57f68e6e5c3bb26f6454deca4e43be13fd0"}
Jul 28 01:33:27.367991 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:27.367951    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd" podStartSLOduration=2.095792171 podStartE2EDuration="5.367935456s" totalImagesPullingTime="3.272143285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:27.365757562 +0000 UTC m=+487.538502807" watchObservedRunningTime="2026-07-28 01:33:27.367935456 +0000 UTC m=+487.540680704"
Jul 28 01:33:27.391177 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:27.391124    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zlv7s" podStartSLOduration=2.391111935 podStartE2EDuration="2.391111935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:27.390275826 +0000 UTC m=+487.563021071" watchObservedRunningTime="2026-07-28 01:33:27.391111935 +0000 UTC m=+487.563857183"
Jul 28 01:33:29.890879 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:29.890847    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:29.992123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:29.992086    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:33.342859 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:33.342831    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5zrnd"
Jul 28 01:33:34.049037 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:34.049004    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:34.992738 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:34.992701    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f8krn"]
Jul 28 01:33:34.995094 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:34.995076    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f8krn"
Jul 28 01:33:34.997708 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:34.997691    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5v7bm\""
Jul 28 01:33:35.007345 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:35.007324    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f8krn"]
Jul 28 01:33:35.125049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:35.125018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/236dcbdc-4e22-4e35-90f2-4fbbe9034c17-bound-sa-token\") pod \"cert-manager-55b4d954d5-f8krn\" (UID: \"236dcbdc-4e22-4e35-90f2-4fbbe9034c17\") " pod="cert-manager/cert-manager-55b4d954d5-f8krn"
Jul 28 01:33:35.125216 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:35.125085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp6x\" (UniqueName: \"kubernetes.io/projected/236dcbdc-4e22-4e35-90f2-4fbbe9034c17-kube-api-access-cvp6x\") pod \"cert-manager-55b4d954d5-f8krn\" (UID: \"236dcbdc-4e22-4e35-90f2-4fbbe9034c17\") " pod="cert-manager/cert-manager-55b4d954d5-f8krn"
Jul 28 01:33:35.225654 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:35.225624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp6x\" (UniqueName: \"kubernetes.io/projected/236dcbdc-4e22-4e35-90f2-4fbbe9034c17-kube-api-access-cvp6x\") pod \"cert-manager-55b4d954d5-f8krn\" (UID: \"236dcbdc-4e22-4e35-90f2-4fbbe9034c17\") " pod="cert-manager/cert-manager-55b4d954d5-f8krn"
Jul 28 01:33:35.225824 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:35.225686    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/236dcbdc-4e22-4e35-90f2-4fbbe9034c17-bound-sa-token\") pod \"cert-manager-55b4d954d5-f8krn\" (UID: \"236dcbdc-4e22-4e35-90f2-4fbbe9034c17\") " pod="cert-manager/cert-manager-55b4d954d5-f8krn"
Jul 28 01:33:35.234665 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:35.234640    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/236dcbdc-4e22-4e35-90f2-4fbbe9034c17-bound-sa-token\") pod \"cert-manager-55b4d954d5-f8krn\" (UID: \"236dcbdc-4e22-4e35-90f2-4fbbe9034c17\") " pod="cert-manager/cert-manager-55b4d954d5-f8krn"
Jul 28 01:33:35.234765 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:35.234703    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp6x\" (UniqueName: \"kubernetes.io/projected/236dcbdc-4e22-4e35-90f2-4fbbe9034c17-kube-api-access-cvp6x\") pod \"cert-manager-55b4d954d5-f8krn\" (UID: \"236dcbdc-4e22-4e35-90f2-4fbbe9034c17\") " pod="cert-manager/cert-manager-55b4d954d5-f8krn"
Jul 28 01:33:35.304066 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:35.303993    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f8krn"
Jul 28 01:33:35.429403 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:35.429377    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f8krn"]
Jul 28 01:33:35.432074 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:33:35.432044    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod236dcbdc_4e22_4e35_90f2_4fbbe9034c17.slice/crio-e01d4181701e47421960832681d104eefab28b55db7bddbf0eedb7806e9ba785 WatchSource:0}: Error finding container e01d4181701e47421960832681d104eefab28b55db7bddbf0eedb7806e9ba785: Status 404 returned error can't find the container with id e01d4181701e47421960832681d104eefab28b55db7bddbf0eedb7806e9ba785
Jul 28 01:33:36.375902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:36.375869    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f8krn" event={"ID":"236dcbdc-4e22-4e35-90f2-4fbbe9034c17","Type":"ContainerStarted","Data":"811290f9fc12899ebb1d00cb0a1fb701b4d62b7224a2d15f0b2628ef2206aec1"}
Jul 28 01:33:36.375902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:36.375906    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f8krn" event={"ID":"236dcbdc-4e22-4e35-90f2-4fbbe9034c17","Type":"ContainerStarted","Data":"e01d4181701e47421960832681d104eefab28b55db7bddbf0eedb7806e9ba785"}
Jul 28 01:33:36.392430 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:36.392383    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-f8krn" podStartSLOduration=2.392367936 podStartE2EDuration="2.392367936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:36.391386938 +0000 UTC m=+496.564132187" watchObservedRunningTime="2026-07-28 01:33:36.392367936 +0000 UTC m=+496.565113185"
Jul 28 01:33:39.117746 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:39.117715    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:39.221603 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:39.221551    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:50.708729 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:50.708687    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:51.168261 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:51.168209    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:51.223035 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:51.223010    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:55.748695 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:55.748662    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:55.849048 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:55.849015    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:56.761365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:56.761318    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:59.572499 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:59.572464    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:59.639068 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:33:59.639037    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:06.843233 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.843200    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-866cdddcb6-mbxzp"]
Jul 28 01:34:06.847770 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.847749    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:06.858008 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.857985    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866cdddcb6-mbxzp"]
Jul 28 01:34:06.889836 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.889813    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:06.986933 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.986903    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-oauth-serving-cert\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:06.987063 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.986949    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-trusted-ca-bundle\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:06.987063 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.986968    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-console-oauth-config\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:06.987063 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.987018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-console-serving-cert\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:06.987063 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.987052    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-service-ca\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:06.987201 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.987079    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-console-config\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:06.987201 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.987094    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5jj\" (UniqueName: \"kubernetes.io/projected/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-kube-api-access-jv5jj\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:06.990788 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:06.990754    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:07.087722 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.087682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-console-config\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.087888 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.087727    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5jj\" (UniqueName: \"kubernetes.io/projected/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-kube-api-access-jv5jj\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.087888 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.087800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-oauth-serving-cert\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.087888 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.087855    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-trusted-ca-bundle\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.087888 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.087883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-console-oauth-config\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.088098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.087911    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-console-serving-cert\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.088098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.087933    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-service-ca\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.088507 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.088481    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-console-config\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.088673 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.088616    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-service-ca\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.088771 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.088741    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-oauth-serving-cert\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.089030 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.089008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-trusted-ca-bundle\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.090571 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.090538    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-console-serving-cert\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.090732 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.090714    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-console-oauth-config\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.095913 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.095863    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5jj\" (UniqueName: \"kubernetes.io/projected/00ee1a01-9842-47cf-ab97-fefd4e1ef0ca-kube-api-access-jv5jj\") pod \"console-866cdddcb6-mbxzp\" (UID: \"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca\") " pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.156920 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.156893    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:07.198777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.198743    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:07.286462 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.286438    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866cdddcb6-mbxzp"]
Jul 28 01:34:07.289131 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:34:07.289107    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ee1a01_9842_47cf_ab97_fefd4e1ef0ca.slice/crio-252d93172026480912b02d27077dfbb20834414f4c2dca37b9d4896e9373a7e7 WatchSource:0}: Error finding container 252d93172026480912b02d27077dfbb20834414f4c2dca37b9d4896e9373a7e7: Status 404 returned error can't find the container with id 252d93172026480912b02d27077dfbb20834414f4c2dca37b9d4896e9373a7e7
Jul 28 01:34:07.289597 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.289543    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:07.498930 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.498848    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866cdddcb6-mbxzp" event={"ID":"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca","Type":"ContainerStarted","Data":"f4e97278e15471f285a1c2f6f7081e387221a289c2296a68a5b0b4f4b5af2d82"}
Jul 28 01:34:07.498930 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.498908    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866cdddcb6-mbxzp" event={"ID":"00ee1a01-9842-47cf-ab97-fefd4e1ef0ca","Type":"ContainerStarted","Data":"252d93172026480912b02d27077dfbb20834414f4c2dca37b9d4896e9373a7e7"}
Jul 28 01:34:07.516885 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:07.516837    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-866cdddcb6-mbxzp" podStartSLOduration=1.516822041 podStartE2EDuration="1.516822041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:07.515771596 +0000 UTC m=+527.688516845" watchObservedRunningTime="2026-07-28 01:34:07.516822041 +0000 UTC m=+527.689567289"
Jul 28 01:34:10.448788 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.448761    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:10.869197 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.869158    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:10.896447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.896421    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"]
Jul 28 01:34:10.898985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.898964    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:10.902737 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.902715    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 01:34:10.902873 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.902753    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:34:10.902873 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.902825    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bw4kj\""
Jul 28 01:34:10.902873 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.902850    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 01:34:10.903107 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.902884    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 01:34:10.903107 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.902856    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 01:34:10.910878 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.910857    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"]
Jul 28 01:34:10.918383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.918365    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/776833e5-d7ec-464b-aa02-611e28dc1c10-manager-config\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:10.918478 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.918393    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/776833e5-d7ec-464b-aa02-611e28dc1c10-cert\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:10.918478 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.918415    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlkb\" (UniqueName: \"kubernetes.io/projected/776833e5-d7ec-464b-aa02-611e28dc1c10-kube-api-access-wqlkb\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:10.918550 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.918491    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/776833e5-d7ec-464b-aa02-611e28dc1c10-metrics-cert\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:10.955536 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:10.955510    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:11.019031 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.019000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/776833e5-d7ec-464b-aa02-611e28dc1c10-metrics-cert\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:11.019146 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.019051    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/776833e5-d7ec-464b-aa02-611e28dc1c10-manager-config\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:11.019146 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.019080    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/776833e5-d7ec-464b-aa02-611e28dc1c10-cert\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:11.019146 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.019103    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlkb\" (UniqueName: \"kubernetes.io/projected/776833e5-d7ec-464b-aa02-611e28dc1c10-kube-api-access-wqlkb\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:11.019789 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.019764    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/776833e5-d7ec-464b-aa02-611e28dc1c10-manager-config\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:11.021610 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.021589    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/776833e5-d7ec-464b-aa02-611e28dc1c10-metrics-cert\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:11.021716 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.021659    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/776833e5-d7ec-464b-aa02-611e28dc1c10-cert\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:11.035676 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.035652    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlkb\" (UniqueName: \"kubernetes.io/projected/776833e5-d7ec-464b-aa02-611e28dc1c10-kube-api-access-wqlkb\") pod \"lws-controller-manager-78db6c7dfd-p54hf\" (UID: \"776833e5-d7ec-464b-aa02-611e28dc1c10\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:11.208611 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.208518    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:11.338170 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.338141    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"]
Jul 28 01:34:11.341616 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:34:11.341584    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776833e5_d7ec_464b_aa02_611e28dc1c10.slice/crio-2ddf463c7d94aafe0e8e6b5c45bbbcf07d682f6feb471344a2a95ad11af220ec WatchSource:0}: Error finding container 2ddf463c7d94aafe0e8e6b5c45bbbcf07d682f6feb471344a2a95ad11af220ec: Status 404 returned error can't find the container with id 2ddf463c7d94aafe0e8e6b5c45bbbcf07d682f6feb471344a2a95ad11af220ec
Jul 28 01:34:11.515970 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:11.515865    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf" event={"ID":"776833e5-d7ec-464b-aa02-611e28dc1c10","Type":"ContainerStarted","Data":"2ddf463c7d94aafe0e8e6b5c45bbbcf07d682f6feb471344a2a95ad11af220ec"}
Jul 28 01:34:13.039106 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:13.039070    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:13.225863 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:13.225827    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:14.531840 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:14.531801    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf" event={"ID":"776833e5-d7ec-464b-aa02-611e28dc1c10","Type":"ContainerStarted","Data":"98835dc4f342e38ee5d656cfc3cfa06ed0ec8149bbf988225a7a748419e3744d"}
Jul 28 01:34:14.532245 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:14.531884    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:14.552542 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:14.552493    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf" podStartSLOduration=2.273339024 podStartE2EDuration="4.552479962s" totalImagesPullingTime="2.279140938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:14.550241959 +0000 UTC m=+534.722987207" watchObservedRunningTime="2026-07-28 01:34:14.552479962 +0000 UTC m=+534.725225286"
Jul 28 01:34:17.157545 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:17.157503    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:17.157946 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:17.157589    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:17.162334 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:17.162312    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:17.548492 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:17.548414    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-866cdddcb6-mbxzp"
Jul 28 01:34:17.600985 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:17.600955    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8d7c6dc9-52jr2"]
Jul 28 01:34:17.713442 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:17.713404    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:17.814506 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:17.814434    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:17.917383 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:17.917354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:25.049066 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:25.049028    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:25.538394 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:25.538364    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-p54hf"
Jul 28 01:34:27.247291 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:27.247258    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:27.626286 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:27.626256    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:28.683052 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:28.683012    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:28.784217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:28.784175    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:29.922242 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:29.922209    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:30.036333 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:30.036306    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:30.632215 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:30.632184    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:31.044687 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:31.044611    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:31.677407 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:31.677377    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:31.807613 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:31.807575    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:34.466011 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:34.465976    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:35.938180 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:35.938140    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"]
Jul 28 01:34:35.941868 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:35.941844    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"
Jul 28 01:34:35.944660 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:35.944626    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 01:34:35.944756 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:35.944658    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 01:34:35.945745 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:35.945730    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wwl84\""
Jul 28 01:34:35.951715 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:35.951693    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"]
Jul 28 01:34:36.011738 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:36.011711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8rlx\" (UniqueName: \"kubernetes.io/projected/57dc03c7-38f0-4018-b784-75857daddee3-kube-api-access-k8rlx\") pod \"limitador-operator-controller-manager-774d94476f-vnhf4\" (UID: \"57dc03c7-38f0-4018-b784-75857daddee3\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"
Jul 28 01:34:36.112930 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:36.112902    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rlx\" (UniqueName: \"kubernetes.io/projected/57dc03c7-38f0-4018-b784-75857daddee3-kube-api-access-k8rlx\") pod \"limitador-operator-controller-manager-774d94476f-vnhf4\" (UID: \"57dc03c7-38f0-4018-b784-75857daddee3\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"
Jul 28 01:34:36.122422 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:36.122400    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rlx\" (UniqueName: \"kubernetes.io/projected/57dc03c7-38f0-4018-b784-75857daddee3-kube-api-access-k8rlx\") pod \"limitador-operator-controller-manager-774d94476f-vnhf4\" (UID: \"57dc03c7-38f0-4018-b784-75857daddee3\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"
Jul 28 01:34:36.252746 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:36.252672    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"
Jul 28 01:34:36.370984 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:36.370953    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:36.388306 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:36.388285    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"]
Jul 28 01:34:36.390616 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:34:36.390586    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dc03c7_38f0_4018_b784_75857daddee3.slice/crio-db5c29deb82c30bdfac20550f50bc854df83ded6595b427167952227869da1f4 WatchSource:0}: Error finding container db5c29deb82c30bdfac20550f50bc854df83ded6595b427167952227869da1f4: Status 404 returned error can't find the container with id db5c29deb82c30bdfac20550f50bc854df83ded6595b427167952227869da1f4
Jul 28 01:34:36.620516 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:36.620484    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4" event={"ID":"57dc03c7-38f0-4018-b784-75857daddee3","Type":"ContainerStarted","Data":"db5c29deb82c30bdfac20550f50bc854df83ded6595b427167952227869da1f4"}
Jul 28 01:34:37.454584 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:37.454520    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:37.556214 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:37.556175    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:39.635975 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:39.635883    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4" event={"ID":"57dc03c7-38f0-4018-b784-75857daddee3","Type":"ContainerStarted","Data":"5e431b5a1ca535d4d7feac33a90ad1d9bef6a8d7bc3b5e8ae2599b56e8851434"}
Jul 28 01:34:39.636343 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:39.636071    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"
Jul 28 01:34:39.656988 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:39.656938    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4" podStartSLOduration=1.715101852 podStartE2EDuration="4.656924134s" totalImagesPullingTime="2.941822282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:39.655858362 +0000 UTC m=+559.828603611" watchObservedRunningTime="2026-07-28 01:34:39.656924134 +0000 UTC m=+559.829669383"
Jul 28 01:34:41.631381 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:41.631347    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:42.060656 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.060578    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:42.162685 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.162657    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:42.263611 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.263574    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:42.363461 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.363429    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:42.468143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.468101    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:42.621776 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.621689    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b8d7c6dc9-52jr2" podUID="1a954194-6a84-49d8-94e8-dd42a9cd8542" containerName="console" containerID="cri-o://b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b" gracePeriod=15
Jul 28 01:34:42.865265 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.865244    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:34:42.979410 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979322    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-serving-cert\") pod \"1a954194-6a84-49d8-94e8-dd42a9cd8542\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") "
Jul 28 01:34:42.979410 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979391    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-service-ca\") pod \"1a954194-6a84-49d8-94e8-dd42a9cd8542\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") "
Jul 28 01:34:42.979675 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979484    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-oauth-serving-cert\") pod \"1a954194-6a84-49d8-94e8-dd42a9cd8542\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") "
Jul 28 01:34:42.979675 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979532    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-trusted-ca-bundle\") pod \"1a954194-6a84-49d8-94e8-dd42a9cd8542\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") "
Jul 28 01:34:42.979785 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979675    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87rbc\" (UniqueName: \"kubernetes.io/projected/1a954194-6a84-49d8-94e8-dd42a9cd8542-kube-api-access-87rbc\") pod \"1a954194-6a84-49d8-94e8-dd42a9cd8542\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") "
Jul 28 01:34:42.979785 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979711    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-oauth-config\") pod \"1a954194-6a84-49d8-94e8-dd42a9cd8542\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") "
Jul 28 01:34:42.979785 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979740    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-config\") pod \"1a954194-6a84-49d8-94e8-dd42a9cd8542\" (UID: \"1a954194-6a84-49d8-94e8-dd42a9cd8542\") "
Jul 28 01:34:42.979932 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979789    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-service-ca" (OuterVolumeSpecName: "service-ca") pod "1a954194-6a84-49d8-94e8-dd42a9cd8542" (UID: "1a954194-6a84-49d8-94e8-dd42a9cd8542"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:34:42.979932 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979868    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1a954194-6a84-49d8-94e8-dd42a9cd8542" (UID: "1a954194-6a84-49d8-94e8-dd42a9cd8542"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:34:42.979932 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.979883    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1a954194-6a84-49d8-94e8-dd42a9cd8542" (UID: "1a954194-6a84-49d8-94e8-dd42a9cd8542"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:34:42.980051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.980009    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-oauth-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:34:42.980051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.980024    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-trusted-ca-bundle\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:34:42.980051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.980037    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-service-ca\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:34:42.980224 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.980205    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-config" (OuterVolumeSpecName: "console-config") pod "1a954194-6a84-49d8-94e8-dd42a9cd8542" (UID: "1a954194-6a84-49d8-94e8-dd42a9cd8542"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:34:42.981696 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.981676    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1a954194-6a84-49d8-94e8-dd42a9cd8542" (UID: "1a954194-6a84-49d8-94e8-dd42a9cd8542"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:34:42.981839 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.981821    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1a954194-6a84-49d8-94e8-dd42a9cd8542" (UID: "1a954194-6a84-49d8-94e8-dd42a9cd8542"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:34:42.981940 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:42.981914    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a954194-6a84-49d8-94e8-dd42a9cd8542-kube-api-access-87rbc" (OuterVolumeSpecName: "kube-api-access-87rbc") pod "1a954194-6a84-49d8-94e8-dd42a9cd8542" (UID: "1a954194-6a84-49d8-94e8-dd42a9cd8542"). InnerVolumeSpecName "kube-api-access-87rbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:34:43.081158 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.081115    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87rbc\" (UniqueName: \"kubernetes.io/projected/1a954194-6a84-49d8-94e8-dd42a9cd8542-kube-api-access-87rbc\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:34:43.081158 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.081153    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-oauth-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:34:43.081158 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.081163    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-config\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:34:43.081158 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.081171    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a954194-6a84-49d8-94e8-dd42a9cd8542-console-serving-cert\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:34:43.654576 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.654523    2556 generic.go:356] "Generic (PLEG): container finished" podID="1a954194-6a84-49d8-94e8-dd42a9cd8542" containerID="b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b" exitCode=0
Jul 28 01:34:43.654752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.654590    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d7c6dc9-52jr2"
Jul 28 01:34:43.654752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.654607    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d7c6dc9-52jr2" event={"ID":"1a954194-6a84-49d8-94e8-dd42a9cd8542","Type":"ContainerDied","Data":"b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b"}
Jul 28 01:34:43.654752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.654661    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d7c6dc9-52jr2" event={"ID":"1a954194-6a84-49d8-94e8-dd42a9cd8542","Type":"ContainerDied","Data":"27a24f023835e3a1ded54bd82cf6ca1ca4ccada0c6931eecc026317d340bbd9e"}
Jul 28 01:34:43.654752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.654681    2556 scope.go:117] "RemoveContainer" containerID="b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b"
Jul 28 01:34:43.663587 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.663544    2556 scope.go:117] "RemoveContainer" containerID="b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b"
Jul 28 01:34:43.663823 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:34:43.663805    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b\": container with ID starting with b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b not found: ID does not exist" containerID="b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b"
Jul 28 01:34:43.663890 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.663834    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b"} err="failed to get container status \"b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b\": rpc error: code = NotFound desc = could not find container \"b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b\": container with ID starting with b5aaaa20463a709bed74749a29bed0a457b27708adbdbbe8fb789ab2fd9f462b not found: ID does not exist"
Jul 28 01:34:43.687449 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.687398    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8d7c6dc9-52jr2"]
Jul 28 01:34:43.690197 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:43.690176    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b8d7c6dc9-52jr2"]
Jul 28 01:34:44.291519 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:44.291483    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a954194-6a84-49d8-94e8-dd42a9cd8542" path="/var/lib/kubelet/pods/1a954194-6a84-49d8-94e8-dd42a9cd8542/volumes"
Jul 28 01:34:47.310162 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:47.310132    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:47.425769 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:47.425726    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:47.543278 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:47.543244    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:50.643486 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:50.643452    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-vnhf4"
Jul 28 01:34:54.281612 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:54.281572    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:54.383643 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:54.383603    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:56.770501 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:56.770472    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:56.869638 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:34:56.869607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:01.426878 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:01.426828    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:01.495788 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:01.495754    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:01.630862 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:01.630826    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:01.731916 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:01.731837    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:03.668194 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:03.668154    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:06.569362 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:06.569322    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:06.669833 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:06.669805    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:13.189205 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:13.189173    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:13.290917 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:13.290857    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:15.789947 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:15.789915    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:15.894792 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:15.894760    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:20.250314 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:20.250286    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:35:20.254081 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:20.254059    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:35:21.179619 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:21.179589    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:21.234224 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:21.234193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:21.576520 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:21.576491    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:21.635722 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:21.635695    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:25.727865 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:25.727823    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:25.852260 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:25.852212    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:25.947834 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:25.947804    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:26.051306 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.051226    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:26.085681 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.085651    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"]
Jul 28 01:35:26.085991 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.085978    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a954194-6a84-49d8-94e8-dd42a9cd8542" containerName="console"
Jul 28 01:35:26.086041 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.085995    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a954194-6a84-49d8-94e8-dd42a9cd8542" containerName="console"
Jul 28 01:35:26.086076 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.086071    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a954194-6a84-49d8-94e8-dd42a9cd8542" containerName="console"
Jul 28 01:35:26.089955 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.089935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:26.092658 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.092634    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 01:35:26.092997 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.092983    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mk4n5\""
Jul 28 01:35:26.099209 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.099189    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"]
Jul 28 01:35:26.124832 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.124806    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe32101c-ed53-4867-8572-0d8016433f47-config-file\") pod \"limitador-limitador-85d9db4d5d-n86fd\" (UID: \"fe32101c-ed53-4867-8572-0d8016433f47\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:26.124925 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.124851    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmdf\" (UniqueName: \"kubernetes.io/projected/fe32101c-ed53-4867-8572-0d8016433f47-kube-api-access-2vmdf\") pod \"limitador-limitador-85d9db4d5d-n86fd\" (UID: \"fe32101c-ed53-4867-8572-0d8016433f47\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:26.131449 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.131430    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"]
Jul 28 01:35:26.225830 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.225804    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmdf\" (UniqueName: \"kubernetes.io/projected/fe32101c-ed53-4867-8572-0d8016433f47-kube-api-access-2vmdf\") pod \"limitador-limitador-85d9db4d5d-n86fd\" (UID: \"fe32101c-ed53-4867-8572-0d8016433f47\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:26.225942 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.225858    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe32101c-ed53-4867-8572-0d8016433f47-config-file\") pod \"limitador-limitador-85d9db4d5d-n86fd\" (UID: \"fe32101c-ed53-4867-8572-0d8016433f47\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:26.226400 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.226384    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe32101c-ed53-4867-8572-0d8016433f47-config-file\") pod \"limitador-limitador-85d9db4d5d-n86fd\" (UID: \"fe32101c-ed53-4867-8572-0d8016433f47\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:26.235154 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.235128    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmdf\" (UniqueName: \"kubernetes.io/projected/fe32101c-ed53-4867-8572-0d8016433f47-kube-api-access-2vmdf\") pod \"limitador-limitador-85d9db4d5d-n86fd\" (UID: \"fe32101c-ed53-4867-8572-0d8016433f47\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:26.400476 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.400441    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:26.548751 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:35:26.548717    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe32101c_ed53_4867_8572_0d8016433f47.slice/crio-0729892444f8616d75a399d2b389de76f05d39be9f3efe1b73d5c1c5e961cf6b WatchSource:0}: Error finding container 0729892444f8616d75a399d2b389de76f05d39be9f3efe1b73d5c1c5e961cf6b: Status 404 returned error can't find the container with id 0729892444f8616d75a399d2b389de76f05d39be9f3efe1b73d5c1c5e961cf6b
Jul 28 01:35:26.559523 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.559500    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"]
Jul 28 01:35:26.648216 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.648174    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:26.763533 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.763453    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:26.818816 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.818785    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vhjw4"]
Jul 28 01:35:26.823265 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.823243    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vhjw4"
Jul 28 01:35:26.826466 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.826445    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tj9kz\""
Jul 28 01:35:26.831396 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.831376    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vhjw4"]
Jul 28 01:35:26.839954 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.839928    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd" event={"ID":"fe32101c-ed53-4867-8572-0d8016433f47","Type":"ContainerStarted","Data":"0729892444f8616d75a399d2b389de76f05d39be9f3efe1b73d5c1c5e961cf6b"}
Jul 28 01:35:26.932578 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:26.932531    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9q6\" (UniqueName: \"kubernetes.io/projected/b4f243f7-45df-4360-9ee1-4dd59d7e0289-kube-api-access-4h9q6\") pod \"authorino-c548b5d7d-vhjw4\" (UID: \"b4f243f7-45df-4360-9ee1-4dd59d7e0289\") " pod="kuadrant-system/authorino-c548b5d7d-vhjw4"
Jul 28 01:35:27.033525 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:27.033432    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9q6\" (UniqueName: \"kubernetes.io/projected/b4f243f7-45df-4360-9ee1-4dd59d7e0289-kube-api-access-4h9q6\") pod \"authorino-c548b5d7d-vhjw4\" (UID: \"b4f243f7-45df-4360-9ee1-4dd59d7e0289\") " pod="kuadrant-system/authorino-c548b5d7d-vhjw4"
Jul 28 01:35:27.043623 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:27.043587    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9q6\" (UniqueName: \"kubernetes.io/projected/b4f243f7-45df-4360-9ee1-4dd59d7e0289-kube-api-access-4h9q6\") pod \"authorino-c548b5d7d-vhjw4\" (UID: \"b4f243f7-45df-4360-9ee1-4dd59d7e0289\") " pod="kuadrant-system/authorino-c548b5d7d-vhjw4"
Jul 28 01:35:27.133489 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:27.133448    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vhjw4"
Jul 28 01:35:27.272076 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:27.272047    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vhjw4"]
Jul 28 01:35:27.476506 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:35:27.476476    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4f243f7_45df_4360_9ee1_4dd59d7e0289.slice/crio-827db2936f30704036260f75136b5995597c43a082db0ddae38b4a1b5ce8d046 WatchSource:0}: Error finding container 827db2936f30704036260f75136b5995597c43a082db0ddae38b4a1b5ce8d046: Status 404 returned error can't find the container with id 827db2936f30704036260f75136b5995597c43a082db0ddae38b4a1b5ce8d046
Jul 28 01:35:27.845168 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:27.845137    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd" event={"ID":"fe32101c-ed53-4867-8572-0d8016433f47","Type":"ContainerStarted","Data":"fa7acd3466361d0be1d9d25b7ec4863081a5d3d05e7b58cd0a3aa060404fdf0b"}
Jul 28 01:35:27.845624 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:27.845249    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:27.846303 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:27.846282    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vhjw4" event={"ID":"b4f243f7-45df-4360-9ee1-4dd59d7e0289","Type":"ContainerStarted","Data":"827db2936f30704036260f75136b5995597c43a082db0ddae38b4a1b5ce8d046"}
Jul 28 01:35:27.861997 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:27.861952    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd" podStartSLOduration=0.880540393 podStartE2EDuration="1.861938914s" totalImagesPullingTime="981.398521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:35:27.861444644 +0000 UTC m=+608.034189892" watchObservedRunningTime="2026-07-28 01:35:27.861938914 +0000 UTC m=+608.034684163"
Jul 28 01:35:29.858996 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:29.858948    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vhjw4" event={"ID":"b4f243f7-45df-4360-9ee1-4dd59d7e0289","Type":"ContainerStarted","Data":"857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33"}
Jul 28 01:35:29.879602 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:29.879527    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-vhjw4" podStartSLOduration=1.60506908 podStartE2EDuration="3.879509941s" totalImagesPullingTime="2.274440861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:35:29.878858473 +0000 UTC m=+610.051603730" watchObservedRunningTime="2026-07-28 01:35:29.879509941 +0000 UTC m=+610.052255196"
Jul 28 01:35:32.861451 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:32.861329    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:32.961925 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:32.961890    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:38.853920 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:38.853884    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-n86fd"
Jul 28 01:35:44.789934 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:44.789895    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:44.890456 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:44.890426    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:44.951197 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:44.951156    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:45.121759 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:45.121724    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:51.579175 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.579146    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vhjw4"]
Jul 28 01:35:51.579549 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.579324    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-vhjw4" podUID="b4f243f7-45df-4360-9ee1-4dd59d7e0289" containerName="authorino" containerID="cri-o://857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33" gracePeriod=30
Jul 28 01:35:51.832960 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.832907    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vhjw4"
Jul 28 01:35:51.945144 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.945113    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h9q6\" (UniqueName: \"kubernetes.io/projected/b4f243f7-45df-4360-9ee1-4dd59d7e0289-kube-api-access-4h9q6\") pod \"b4f243f7-45df-4360-9ee1-4dd59d7e0289\" (UID: \"b4f243f7-45df-4360-9ee1-4dd59d7e0289\") "
Jul 28 01:35:51.947400 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.947376    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f243f7-45df-4360-9ee1-4dd59d7e0289-kube-api-access-4h9q6" (OuterVolumeSpecName: "kube-api-access-4h9q6") pod "b4f243f7-45df-4360-9ee1-4dd59d7e0289" (UID: "b4f243f7-45df-4360-9ee1-4dd59d7e0289"). InnerVolumeSpecName "kube-api-access-4h9q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:35:51.955999 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.955970    2556 generic.go:356] "Generic (PLEG): container finished" podID="b4f243f7-45df-4360-9ee1-4dd59d7e0289" containerID="857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33" exitCode=0
Jul 28 01:35:51.956091 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.956013    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vhjw4"
Jul 28 01:35:51.956134 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.956011    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vhjw4" event={"ID":"b4f243f7-45df-4360-9ee1-4dd59d7e0289","Type":"ContainerDied","Data":"857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33"}
Jul 28 01:35:51.956176 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.956133    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vhjw4" event={"ID":"b4f243f7-45df-4360-9ee1-4dd59d7e0289","Type":"ContainerDied","Data":"827db2936f30704036260f75136b5995597c43a082db0ddae38b4a1b5ce8d046"}
Jul 28 01:35:51.956176 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.956158    2556 scope.go:117] "RemoveContainer" containerID="857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33"
Jul 28 01:35:51.964680 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.964665    2556 scope.go:117] "RemoveContainer" containerID="857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33"
Jul 28 01:35:51.964977 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:35:51.964951    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33\": container with ID starting with 857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33 not found: ID does not exist" containerID="857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33"
Jul 28 01:35:51.965051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.964984    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33"} err="failed to get container status \"857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33\": rpc error: code = NotFound desc = could not find container \"857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33\": container with ID starting with 857b76e7b55ceb343edfb8fac3a9c5163d8340334415080768965ccd7614dd33 not found: ID does not exist"
Jul 28 01:35:51.976961 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.976938    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vhjw4"]
Jul 28 01:35:51.980836 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:51.980817    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vhjw4"]
Jul 28 01:35:52.046281 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:52.046252    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h9q6\" (UniqueName: \"kubernetes.io/projected/b4f243f7-45df-4360-9ee1-4dd59d7e0289-kube-api-access-4h9q6\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:35:52.287365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:52.287296    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f243f7-45df-4360-9ee1-4dd59d7e0289" path="/var/lib/kubelet/pods/b4f243f7-45df-4360-9ee1-4dd59d7e0289/volumes"
Jul 28 01:35:55.741078 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:55.741046    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:55.815511 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:35:55.815474    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:08.303106 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:36:08.303076    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:17.400967 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:36:17.400936    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:34.612735 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:36:34.612702    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:34.731834 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:36:34.731801    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:34.892752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:36:34.892667    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:34.990724 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:36:34.990693    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:35.331708 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:36:35.331673    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:35.433367 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:36:35.433335    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:23.436863 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:37:23.436779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:23.537631 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:37:23.537600    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:10.700669 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:10.700634    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:10.796474 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:10.796443    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:22.939945 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:22.939910    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:23.035925 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:23.035889    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:26.887179 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:26.887123    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:51.617782 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.617746    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:51.857160 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.857133    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"]
Jul 28 01:38:51.857489 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.857476    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f243f7-45df-4360-9ee1-4dd59d7e0289" containerName="authorino"
Jul 28 01:38:51.857540 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.857493    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f243f7-45df-4360-9ee1-4dd59d7e0289" containerName="authorino"
Jul 28 01:38:51.857607 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.857600    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f243f7-45df-4360-9ee1-4dd59d7e0289" containerName="authorino"
Jul 28 01:38:51.860752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.860734    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:38:51.864969 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.864949    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-qt4xs\""
Jul 28 01:38:51.865079 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.864977    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 28 01:38:51.865079 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.864989    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-swjq9\""
Jul 28 01:38:51.865079 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.864952    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 28 01:38:51.865249 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.865084    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 28 01:38:51.869755 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.869713    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"]
Jul 28 01:38:51.889934 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:51.889910    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:52.040866 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.040825    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009a349a-bea9-4d8c-8826-f1b4aa59411a-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n\" (UID: \"009a349a-bea9-4d8c-8826-f1b4aa59411a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:38:52.041042 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.040901    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcl7p\" (UniqueName: \"kubernetes.io/projected/009a349a-bea9-4d8c-8826-f1b4aa59411a-kube-api-access-bcl7p\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n\" (UID: \"009a349a-bea9-4d8c-8826-f1b4aa59411a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:38:52.141649 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.141534    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcl7p\" (UniqueName: \"kubernetes.io/projected/009a349a-bea9-4d8c-8826-f1b4aa59411a-kube-api-access-bcl7p\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n\" (UID: \"009a349a-bea9-4d8c-8826-f1b4aa59411a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:38:52.141797 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.141678    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009a349a-bea9-4d8c-8826-f1b4aa59411a-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n\" (UID: \"009a349a-bea9-4d8c-8826-f1b4aa59411a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:38:52.144310 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.144286    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009a349a-bea9-4d8c-8826-f1b4aa59411a-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n\" (UID: \"009a349a-bea9-4d8c-8826-f1b4aa59411a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:38:52.151161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.151140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcl7p\" (UniqueName: \"kubernetes.io/projected/009a349a-bea9-4d8c-8826-f1b4aa59411a-kube-api-access-bcl7p\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n\" (UID: \"009a349a-bea9-4d8c-8826-f1b4aa59411a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:38:52.172128 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.172108    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:38:52.304022 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.303984    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"]
Jul 28 01:38:52.306949 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:38:52.306912    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009a349a_bea9_4d8c_8826_f1b4aa59411a.slice/crio-8e7cf8d0429fa09c4378a76d8aef093fe58dce99f5361b759b52bac515997534 WatchSource:0}: Error finding container 8e7cf8d0429fa09c4378a76d8aef093fe58dce99f5361b759b52bac515997534: Status 404 returned error can't find the container with id 8e7cf8d0429fa09c4378a76d8aef093fe58dce99f5361b759b52bac515997534
Jul 28 01:38:52.308802 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.308786    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:38:52.723327 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:52.723295    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n" event={"ID":"009a349a-bea9-4d8c-8826-f1b4aa59411a","Type":"ContainerStarted","Data":"8e7cf8d0429fa09c4378a76d8aef093fe58dce99f5361b759b52bac515997534"}
Jul 28 01:38:55.740048 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:55.740014    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n" event={"ID":"009a349a-bea9-4d8c-8826-f1b4aa59411a","Type":"ContainerStarted","Data":"e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f"}
Jul 28 01:38:55.740438 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:55.740157    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:38:55.757867 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:38:55.757821    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n" podStartSLOduration=2.343427036 podStartE2EDuration="4.757805673s" totalImagesPullingTime="2.414378637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:55.756187814 +0000 UTC m=+815.928933063" watchObservedRunningTime="2026-07-28 01:38:55.757805673 +0000 UTC m=+815.930550922"
Jul 28 01:39:08.008227 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:08.008189    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:12.659760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:12.659722    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:26.746185 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:26.746150    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:39:33.709051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:33.709016    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:33.709665 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:33.709639    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"]
Jul 28 01:39:33.764816 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:33.764784    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:36.745551 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:36.745506    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n" podUID="009a349a-bea9-4d8c-8826-f1b4aa59411a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:39:46.746166 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:46.746126    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n" podUID="009a349a-bea9-4d8c-8826-f1b4aa59411a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:39:48.730298 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:48.730230    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n" podUID="009a349a-bea9-4d8c-8826-f1b4aa59411a" containerName="main" containerID="cri-o://e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f" gracePeriod=45
Jul 28 01:39:48.969607 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:48.969584    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:39:48.971131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:48.971108    2556 generic.go:356] "Generic (PLEG): container finished" podID="009a349a-bea9-4d8c-8826-f1b4aa59411a" containerID="e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f" exitCode=0
Jul 28 01:39:48.971246 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:48.971147    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"
Jul 28 01:39:48.971246 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:48.971187    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n" event={"ID":"009a349a-bea9-4d8c-8826-f1b4aa59411a","Type":"ContainerDied","Data":"e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f"}
Jul 28 01:39:48.971317 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:48.971279    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n" event={"ID":"009a349a-bea9-4d8c-8826-f1b4aa59411a","Type":"ContainerDied","Data":"8e7cf8d0429fa09c4378a76d8aef093fe58dce99f5361b759b52bac515997534"}
Jul 28 01:39:48.971317 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:48.971307    2556 scope.go:117] "RemoveContainer" containerID="e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f"
Jul 28 01:39:48.979045 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:48.979027    2556 scope.go:117] "RemoveContainer" containerID="e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f"
Jul 28 01:39:48.979293 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:39:48.979275    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f\": container with ID starting with e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f not found: ID does not exist" containerID="e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f"
Jul 28 01:39:48.979340 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:48.979303    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f"} err="failed to get container status \"e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f\": rpc error: code = NotFound desc = could not find container \"e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f\": container with ID starting with e895a50f7cfd5ab0992ea36c83220d69e406b6356baed8bbb37d8d88cc3d882f not found: ID does not exist"
Jul 28 01:39:49.124737 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:49.124707    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009a349a-bea9-4d8c-8826-f1b4aa59411a-tls-certs\") pod \"009a349a-bea9-4d8c-8826-f1b4aa59411a\" (UID: \"009a349a-bea9-4d8c-8826-f1b4aa59411a\") "
Jul 28 01:39:49.124913 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:49.124805    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcl7p\" (UniqueName: \"kubernetes.io/projected/009a349a-bea9-4d8c-8826-f1b4aa59411a-kube-api-access-bcl7p\") pod \"009a349a-bea9-4d8c-8826-f1b4aa59411a\" (UID: \"009a349a-bea9-4d8c-8826-f1b4aa59411a\") "
Jul 28 01:39:49.127110 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:49.127076    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009a349a-bea9-4d8c-8826-f1b4aa59411a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "009a349a-bea9-4d8c-8826-f1b4aa59411a" (UID: "009a349a-bea9-4d8c-8826-f1b4aa59411a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:39:49.127110 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:49.127094    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009a349a-bea9-4d8c-8826-f1b4aa59411a-kube-api-access-bcl7p" (OuterVolumeSpecName: "kube-api-access-bcl7p") pod "009a349a-bea9-4d8c-8826-f1b4aa59411a" (UID: "009a349a-bea9-4d8c-8826-f1b4aa59411a"). InnerVolumeSpecName "kube-api-access-bcl7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:49.225320 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:49.225288    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009a349a-bea9-4d8c-8826-f1b4aa59411a-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:39:49.225320 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:49.225315    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcl7p\" (UniqueName: \"kubernetes.io/projected/009a349a-bea9-4d8c-8826-f1b4aa59411a-kube-api-access-bcl7p\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:39:49.293575 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:49.293532    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"]
Jul 28 01:39:49.299636 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:49.299610    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mjd7n"]
Jul 28 01:39:50.285838 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:50.285805    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="009a349a-bea9-4d8c-8826-f1b4aa59411a" path="/var/lib/kubelet/pods/009a349a-bea9-4d8c-8826-f1b4aa59411a/volumes"
Jul 28 01:39:53.729728 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:53.729695    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:53.828379 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:53.828344    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:57.230645 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.230610    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"]
Jul 28 01:39:57.231013 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.230926    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009a349a-bea9-4d8c-8826-f1b4aa59411a" containerName="main"
Jul 28 01:39:57.231013 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.230937    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="009a349a-bea9-4d8c-8826-f1b4aa59411a" containerName="main"
Jul 28 01:39:57.231013 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.231000    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="009a349a-bea9-4d8c-8826-f1b4aa59411a" containerName="main"
Jul 28 01:39:57.235182 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.235163    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.238805 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.238784    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 28 01:39:57.238918 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.238784    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 28 01:39:57.238918 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.238785    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 28 01:39:57.238918 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.238785    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-n4fj4\""
Jul 28 01:39:57.244451 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.244429    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"]
Jul 28 01:39:57.290012 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.289986    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-home\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.290145 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.290044    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.290193 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.290133    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.290193 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.290176    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.290261 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.290207    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csn7k\" (UniqueName: \"kubernetes.io/projected/f24f8956-1a3c-4f89-8a52-776c5f3e782b-kube-api-access-csn7k\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.290261 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.290255    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.324542 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.324509    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:57.391645 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.391608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-home\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.391760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.391661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.391760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.391696    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.391760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.391711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.391760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.391728    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csn7k\" (UniqueName: \"kubernetes.io/projected/f24f8956-1a3c-4f89-8a52-776c5f3e782b-kube-api-access-csn7k\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.391760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.391756    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.392080 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.392061    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-home\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.392163 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.392087    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.392163 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.392129    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.394133 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.394113    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.394235 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.394220    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.400129 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.400105    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csn7k\" (UniqueName: \"kubernetes.io/projected/f24f8956-1a3c-4f89-8a52-776c5f3e782b-kube-api-access-csn7k\") pod \"fc-concurrency-test-kserve-b5d44467c-gzsps\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.474690 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.474656    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"]
Jul 28 01:39:57.478176 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.478159    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:39:57.480743 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.480679    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-68qmv\""
Jul 28 01:39:57.488936 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.488913    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"]
Jul 28 01:39:57.493131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.493106    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzv99\" (UniqueName: \"kubernetes.io/projected/33744f57-f0e5-4cf7-963d-00d2ec53ec12-kube-api-access-xzv99\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts\" (UID: \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:39:57.493257 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.493171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33744f57-f0e5-4cf7-963d-00d2ec53ec12-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts\" (UID: \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:39:57.512696 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.512668    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:57.545013 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.544982    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:39:57.593928 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.593885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzv99\" (UniqueName: \"kubernetes.io/projected/33744f57-f0e5-4cf7-963d-00d2ec53ec12-kube-api-access-xzv99\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts\" (UID: \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:39:57.594075 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.594001    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33744f57-f0e5-4cf7-963d-00d2ec53ec12-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts\" (UID: \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:39:57.597028 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.596978    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33744f57-f0e5-4cf7-963d-00d2ec53ec12-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts\" (UID: \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:39:57.608199 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.608170    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzv99\" (UniqueName: \"kubernetes.io/projected/33744f57-f0e5-4cf7-963d-00d2ec53ec12-kube-api-access-xzv99\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts\" (UID: \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:39:57.674608 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.674576    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"]
Jul 28 01:39:57.677546 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:39:57.677512    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24f8956_1a3c_4f89_8a52_776c5f3e782b.slice/crio-12b477bce4f53fafb8aff499b4694d04574adb48250aadf0aade17c4974bce60 WatchSource:0}: Error finding container 12b477bce4f53fafb8aff499b4694d04574adb48250aadf0aade17c4974bce60: Status 404 returned error can't find the container with id 12b477bce4f53fafb8aff499b4694d04574adb48250aadf0aade17c4974bce60
Jul 28 01:39:57.789229 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.789156    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:39:57.925345 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:57.925318    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"]
Jul 28 01:39:57.927640 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:39:57.927613    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33744f57_f0e5_4cf7_963d_00d2ec53ec12.slice/crio-96c35604d4ab661f4bb01400d00eea82a402ab7231783515b210994caadbc68c WatchSource:0}: Error finding container 96c35604d4ab661f4bb01400d00eea82a402ab7231783515b210994caadbc68c: Status 404 returned error can't find the container with id 96c35604d4ab661f4bb01400d00eea82a402ab7231783515b210994caadbc68c
Jul 28 01:39:58.016696 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:58.016658    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts" event={"ID":"33744f57-f0e5-4cf7-963d-00d2ec53ec12","Type":"ContainerStarted","Data":"d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938"}
Jul 28 01:39:58.016901 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:58.016722    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:39:58.016901 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:58.016749    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts" event={"ID":"33744f57-f0e5-4cf7-963d-00d2ec53ec12","Type":"ContainerStarted","Data":"96c35604d4ab661f4bb01400d00eea82a402ab7231783515b210994caadbc68c"}
Jul 28 01:39:58.017884 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:58.017861    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps" event={"ID":"f24f8956-1a3c-4f89-8a52-776c5f3e782b","Type":"ContainerStarted","Data":"12b477bce4f53fafb8aff499b4694d04574adb48250aadf0aade17c4974bce60"}
Jul 28 01:39:58.036123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:39:58.036079    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts" podStartSLOduration=1.036064372 podStartE2EDuration="1.036064372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:39:58.035014469 +0000 UTC m=+878.207759719" watchObservedRunningTime="2026-07-28 01:39:58.036064372 +0000 UTC m=+878.208809634"
Jul 28 01:40:00.029458 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:00.029422    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps" event={"ID":"f24f8956-1a3c-4f89-8a52-776c5f3e782b","Type":"ContainerStarted","Data":"6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d"}
Jul 28 01:40:00.050213 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:00.050139    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps" podStartSLOduration=1.6018010440000001 podStartE2EDuration="3.050118715s" totalImagesPullingTime="1.448317671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:40:00.048614399 +0000 UTC m=+880.221359648" watchObservedRunningTime="2026-07-28 01:40:00.050118715 +0000 UTC m=+880.222863964"
Jul 28 01:40:00.267679 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:00.267646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:00.545659 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:00.545626    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:40:05.962757 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:05.962721    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:06.082933 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:06.082894    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:07.545231 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:07.545197    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:40:07.557796 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:07.557773    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:40:08.075521 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:08.075495    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:40:08.537908 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:08.537825    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:20.280437 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:20.280371    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:40:20.280437 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:20.280417    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:40:29.025649 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:29.025618    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:40:32.025888 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:32.025853    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:32.126498 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:32.126467    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:35.675079 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:35.675046    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"]
Jul 28 01:40:35.677112 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:35.676750    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:35.681889 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:35.681858    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"]
Jul 28 01:40:35.684989 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:40:35.684961    2556 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:40:35.684989 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:40:35.684989 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:40:35.684989 ip-10-0-140-37 kubenswrapper[2556]:  > containerID="6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d" cmd=["/bin/sleep","15"]
Jul 28 01:40:35.685208 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:35.685010    2556 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps" message="" err=<
Jul 28 01:40:35.685208 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:40:35.685208 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:40:35.685208 ip-10-0-140-37 kubenswrapper[2556]:  >
Jul 28 01:40:35.685208 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:40:35.685023    2556 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:40:35.685208 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:40:35.685208 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:40:35.685208 ip-10-0-140-37 kubenswrapper[2556]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps" podUID="f24f8956-1a3c-4f89-8a52-776c5f3e782b" containerName="main" containerID="cri-o://6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d"
Jul 28 01:40:35.685208 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:35.685071    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps" podUID="f24f8956-1a3c-4f89-8a52-776c5f3e782b" containerName="main" containerID="cri-o://6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d" gracePeriod=60
Jul 28 01:40:35.742974 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:35.742945    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:36.034100 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.034078    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:40:36.120978 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.120947    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-dshm\") pod \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") "
Jul 28 01:40:36.120978 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.120983    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-home\") pod \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") "
Jul 28 01:40:36.121208 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.121034    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csn7k\" (UniqueName: \"kubernetes.io/projected/f24f8956-1a3c-4f89-8a52-776c5f3e782b-kube-api-access-csn7k\") pod \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") "
Jul 28 01:40:36.121208 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.121049    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-model-cache\") pod \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") "
Jul 28 01:40:36.121208 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.121143    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tls-certs\") pod \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") "
Jul 28 01:40:36.121208 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.121169    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tmp-dir\") pod \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\" (UID: \"f24f8956-1a3c-4f89-8a52-776c5f3e782b\") "
Jul 28 01:40:36.121440 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.121306    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-home" (OuterVolumeSpecName: "home") pod "f24f8956-1a3c-4f89-8a52-776c5f3e782b" (UID: "f24f8956-1a3c-4f89-8a52-776c5f3e782b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:36.121440 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.121376    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f24f8956-1a3c-4f89-8a52-776c5f3e782b" (UID: "f24f8956-1a3c-4f89-8a52-776c5f3e782b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:36.121440 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.121419    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-home\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.121619 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.121441    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f24f8956-1a3c-4f89-8a52-776c5f3e782b" (UID: "f24f8956-1a3c-4f89-8a52-776c5f3e782b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:36.123494 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.123462    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-dshm" (OuterVolumeSpecName: "dshm") pod "f24f8956-1a3c-4f89-8a52-776c5f3e782b" (UID: "f24f8956-1a3c-4f89-8a52-776c5f3e782b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:36.123622 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.123495    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24f8956-1a3c-4f89-8a52-776c5f3e782b-kube-api-access-csn7k" (OuterVolumeSpecName: "kube-api-access-csn7k") pod "f24f8956-1a3c-4f89-8a52-776c5f3e782b" (UID: "f24f8956-1a3c-4f89-8a52-776c5f3e782b"). InnerVolumeSpecName "kube-api-access-csn7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:40:36.123622 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.123506    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f24f8956-1a3c-4f89-8a52-776c5f3e782b" (UID: "f24f8956-1a3c-4f89-8a52-776c5f3e782b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:40:36.188777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.188749    2556 generic.go:356] "Generic (PLEG): container finished" podID="f24f8956-1a3c-4f89-8a52-776c5f3e782b" containerID="6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d" exitCode=0
Jul 28 01:40:36.188893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.188809    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"
Jul 28 01:40:36.188893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.188837    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps" event={"ID":"f24f8956-1a3c-4f89-8a52-776c5f3e782b","Type":"ContainerDied","Data":"6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d"}
Jul 28 01:40:36.188964 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.188896    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps" event={"ID":"f24f8956-1a3c-4f89-8a52-776c5f3e782b","Type":"ContainerDied","Data":"12b477bce4f53fafb8aff499b4694d04574adb48250aadf0aade17c4974bce60"}
Jul 28 01:40:36.188964 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.188925    2556 scope.go:117] "RemoveContainer" containerID="6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d"
Jul 28 01:40:36.198360 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.198346    2556 scope.go:117] "RemoveContainer" containerID="6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d"
Jul 28 01:40:36.198639 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:40:36.198617    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d\": container with ID starting with 6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d not found: ID does not exist" containerID="6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d"
Jul 28 01:40:36.198701 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.198651    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d"} err="failed to get container status \"6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d\": rpc error: code = NotFound desc = could not find container \"6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d\": container with ID starting with 6a3383fadbd6e59fe446093bd2dd5ec44ee2489294c28d0995d02b41953ec31d not found: ID does not exist"
Jul 28 01:40:36.210354 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.210300    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"]
Jul 28 01:40:36.213785 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.213764    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-gzsps"]
Jul 28 01:40:36.222584 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.222544    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.222584 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.222578    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-tmp-dir\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.222584 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.222588    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-dshm\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.222738 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.222596    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csn7k\" (UniqueName: \"kubernetes.io/projected/f24f8956-1a3c-4f89-8a52-776c5f3e782b-kube-api-access-csn7k\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.222738 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.222606    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f24f8956-1a3c-4f89-8a52-776c5f3e782b-model-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:40:36.287174 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:36.287148    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f24f8956-1a3c-4f89-8a52-776c5f3e782b" path="/var/lib/kubelet/pods/f24f8956-1a3c-4f89-8a52-776c5f3e782b/volumes"
Jul 28 01:40:39.025438 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:39.025404    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts" podUID="33744f57-f0e5-4cf7-963d-00d2ec53ec12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:40:49.024964 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:49.024923    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts" podUID="33744f57-f0e5-4cf7-963d-00d2ec53ec12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:40:50.701359 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:50.701302    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts" podUID="33744f57-f0e5-4cf7-963d-00d2ec53ec12" containerName="main" containerID="cri-o://d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938" gracePeriod=45
Jul 28 01:40:50.947088 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:50.947067    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:40:51.045520 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.045439    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzv99\" (UniqueName: \"kubernetes.io/projected/33744f57-f0e5-4cf7-963d-00d2ec53ec12-kube-api-access-xzv99\") pod \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\" (UID: \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\") "
Jul 28 01:40:51.045520 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.045486    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33744f57-f0e5-4cf7-963d-00d2ec53ec12-tls-certs\") pod \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\" (UID: \"33744f57-f0e5-4cf7-963d-00d2ec53ec12\") "
Jul 28 01:40:51.047513 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.047480    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33744f57-f0e5-4cf7-963d-00d2ec53ec12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33744f57-f0e5-4cf7-963d-00d2ec53ec12" (UID: "33744f57-f0e5-4cf7-963d-00d2ec53ec12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:40:51.047513 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.047507    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33744f57-f0e5-4cf7-963d-00d2ec53ec12-kube-api-access-xzv99" (OuterVolumeSpecName: "kube-api-access-xzv99") pod "33744f57-f0e5-4cf7-963d-00d2ec53ec12" (UID: "33744f57-f0e5-4cf7-963d-00d2ec53ec12"). InnerVolumeSpecName "kube-api-access-xzv99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:40:51.146117 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.146086    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzv99\" (UniqueName: \"kubernetes.io/projected/33744f57-f0e5-4cf7-963d-00d2ec53ec12-kube-api-access-xzv99\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:40:51.146117 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.146112    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33744f57-f0e5-4cf7-963d-00d2ec53ec12-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:40:51.257853 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.257824    2556 generic.go:356] "Generic (PLEG): container finished" podID="33744f57-f0e5-4cf7-963d-00d2ec53ec12" containerID="d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938" exitCode=0
Jul 28 01:40:51.257989 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.257890    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"
Jul 28 01:40:51.257989 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.257914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts" event={"ID":"33744f57-f0e5-4cf7-963d-00d2ec53ec12","Type":"ContainerDied","Data":"d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938"}
Jul 28 01:40:51.257989 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.257978    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts" event={"ID":"33744f57-f0e5-4cf7-963d-00d2ec53ec12","Type":"ContainerDied","Data":"96c35604d4ab661f4bb01400d00eea82a402ab7231783515b210994caadbc68c"}
Jul 28 01:40:51.258092 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.258012    2556 scope.go:117] "RemoveContainer" containerID="d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938"
Jul 28 01:40:51.266939 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.266921    2556 scope.go:117] "RemoveContainer" containerID="d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938"
Jul 28 01:40:51.267191 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:40:51.267173    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938\": container with ID starting with d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938 not found: ID does not exist" containerID="d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938"
Jul 28 01:40:51.267259 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.267202    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938"} err="failed to get container status \"d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938\": rpc error: code = NotFound desc = could not find container \"d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938\": container with ID starting with d5da953c0313bc303e5c0a6389c504619cda08b116ee1190c604629165d58938 not found: ID does not exist"
Jul 28 01:40:51.283500 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.283481    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"]
Jul 28 01:40:51.290824 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:51.290804    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-qbkts"]
Jul 28 01:40:52.286092 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:52.286059    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33744f57-f0e5-4cf7-963d-00d2ec53ec12" path="/var/lib/kubelet/pods/33744f57-f0e5-4cf7-963d-00d2ec53ec12/volumes"
Jul 28 01:40:55.613664 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:55.613637    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:55.716779 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:55.716744    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:55.827499 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:55.827448    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:55.916927 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:55.916893    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:59.512493 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:40:59.512446    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:02.116122 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:02.116069    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:03.321107 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.321061    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"]
Jul 28 01:41:03.321682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.321597    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33744f57-f0e5-4cf7-963d-00d2ec53ec12" containerName="main"
Jul 28 01:41:03.321682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.321617    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="33744f57-f0e5-4cf7-963d-00d2ec53ec12" containerName="main"
Jul 28 01:41:03.321682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.321630    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f24f8956-1a3c-4f89-8a52-776c5f3e782b" containerName="main"
Jul 28 01:41:03.321682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.321638    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f24f8956-1a3c-4f89-8a52-776c5f3e782b" containerName="main"
Jul 28 01:41:03.321909 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.321724    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f24f8956-1a3c-4f89-8a52-776c5f3e782b" containerName="main"
Jul 28 01:41:03.321909 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.321738    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="33744f57-f0e5-4cf7-963d-00d2ec53ec12" containerName="main"
Jul 28 01:41:03.326828 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.326807    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.330518 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.330493    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 28 01:41:03.330652 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.330519    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 28 01:41:03.330652 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.330519    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 28 01:41:03.330652 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.330521    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-hbgpq\""
Jul 28 01:41:03.333646 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.333614    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"]
Jul 28 01:41:03.424423 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.424378    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:03.464457 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.464425    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnt97\" (UniqueName: \"kubernetes.io/projected/7e5576ad-5eed-4544-9213-f5d820c5f80f-kube-api-access-nnt97\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.464637 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.464486    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.464637 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.464511    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.464637 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.464544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5576ad-5eed-4544-9213-f5d820c5f80f-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.464637 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.464628    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.464957 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.464733    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.565760 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.565726    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5576ad-5eed-4544-9213-f5d820c5f80f-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.565914 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.565766    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.565914 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.565846    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.565914 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.565889    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnt97\" (UniqueName: \"kubernetes.io/projected/7e5576ad-5eed-4544-9213-f5d820c5f80f-kube-api-access-nnt97\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.566071 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.565922    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.566071 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.565945    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.566315 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.566285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.566315 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.566301    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.566477 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.566352    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.568145 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.568126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.568406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.568390    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5576ad-5eed-4544-9213-f5d820c5f80f-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.574009 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.573952    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnt97\" (UniqueName: \"kubernetes.io/projected/7e5576ad-5eed-4544-9213-f5d820c5f80f-kube-api-access-nnt97\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.638051 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.638016    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:03.766369 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:03.766330    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"]
Jul 28 01:41:03.768657 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:41:03.768627    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5576ad_5eed_4544_9213_f5d820c5f80f.slice/crio-c30903c01be27d0dc41c4d73b751871b6c9fd38615f82fc2a22e87ed626b4621 WatchSource:0}: Error finding container c30903c01be27d0dc41c4d73b751871b6c9fd38615f82fc2a22e87ed626b4621: Status 404 returned error can't find the container with id c30903c01be27d0dc41c4d73b751871b6c9fd38615f82fc2a22e87ed626b4621
Jul 28 01:41:04.314451 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:04.314421    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797" event={"ID":"7e5576ad-5eed-4544-9213-f5d820c5f80f","Type":"ContainerStarted","Data":"47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275"}
Jul 28 01:41:04.314645 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:04.314469    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797" event={"ID":"7e5576ad-5eed-4544-9213-f5d820c5f80f","Type":"ContainerStarted","Data":"c30903c01be27d0dc41c4d73b751871b6c9fd38615f82fc2a22e87ed626b4621"}
Jul 28 01:41:04.331674 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:04.331628    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797" podStartSLOduration=1.331612066 podStartE2EDuration="1.331612066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:04.330752862 +0000 UTC m=+944.503498139" watchObservedRunningTime="2026-07-28 01:41:04.331612066 +0000 UTC m=+944.504357321"
Jul 28 01:41:04.638470 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:04.638436    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:05.259012 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:05.258977    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:05.370204 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:05.370174    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:05.469693 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:05.469655    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:05.631461 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:05.631425    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:05.704091 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:05.704065    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:06.617377 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:06.617343    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"]
Jul 28 01:41:06.636955 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:06.636927    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:07.337767 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:41:07.337734    2556 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:41:07.337767 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:41:07.337767 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:41:07.337767 ip-10-0-140-37 kubenswrapper[2556]:  > containerID="47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275" cmd=["/bin/sleep","15"]
Jul 28 01:41:07.338080 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.337779    2556 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797" message="" err=<
Jul 28 01:41:07.338080 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:41:07.338080 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:41:07.338080 ip-10-0-140-37 kubenswrapper[2556]:  >
Jul 28 01:41:07.338080 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:41:07.337791    2556 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:41:07.338080 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:41:07.338080 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:41:07.338080 ip-10-0-140-37 kubenswrapper[2556]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797" podUID="7e5576ad-5eed-4544-9213-f5d820c5f80f" containerName="main" containerID="cri-o://47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275"
Jul 28 01:41:07.338080 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.337841    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797" podUID="7e5576ad-5eed-4544-9213-f5d820c5f80f" containerName="main" containerID="cri-o://47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275" gracePeriod=60
Jul 28 01:41:07.584983 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.584954    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:07.703135 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703060    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-model-cache\") pod \"7e5576ad-5eed-4544-9213-f5d820c5f80f\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") "
Jul 28 01:41:07.703135 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703129    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-home\") pod \"7e5576ad-5eed-4544-9213-f5d820c5f80f\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") "
Jul 28 01:41:07.703672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703162    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnt97\" (UniqueName: \"kubernetes.io/projected/7e5576ad-5eed-4544-9213-f5d820c5f80f-kube-api-access-nnt97\") pod \"7e5576ad-5eed-4544-9213-f5d820c5f80f\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") "
Jul 28 01:41:07.703672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703216    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-dshm\") pod \"7e5576ad-5eed-4544-9213-f5d820c5f80f\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") "
Jul 28 01:41:07.703672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703256    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5576ad-5eed-4544-9213-f5d820c5f80f-tls-certs\") pod \"7e5576ad-5eed-4544-9213-f5d820c5f80f\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") "
Jul 28 01:41:07.703672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703320    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e5576ad-5eed-4544-9213-f5d820c5f80f" (UID: "7e5576ad-5eed-4544-9213-f5d820c5f80f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:07.703672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703357    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-home" (OuterVolumeSpecName: "home") pod "7e5576ad-5eed-4544-9213-f5d820c5f80f" (UID: "7e5576ad-5eed-4544-9213-f5d820c5f80f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:07.703672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703365    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-tmp-dir\") pod \"7e5576ad-5eed-4544-9213-f5d820c5f80f\" (UID: \"7e5576ad-5eed-4544-9213-f5d820c5f80f\") "
Jul 28 01:41:07.703672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703522    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e5576ad-5eed-4544-9213-f5d820c5f80f" (UID: "7e5576ad-5eed-4544-9213-f5d820c5f80f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:07.703672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703663    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-model-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:07.703672 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703677    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-home\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:07.703995 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.703704    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-tmp-dir\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:07.705536 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.705495    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-dshm" (OuterVolumeSpecName: "dshm") pod "7e5576ad-5eed-4544-9213-f5d820c5f80f" (UID: "7e5576ad-5eed-4544-9213-f5d820c5f80f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:07.705678 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.705524    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5576ad-5eed-4544-9213-f5d820c5f80f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e5576ad-5eed-4544-9213-f5d820c5f80f" (UID: "7e5576ad-5eed-4544-9213-f5d820c5f80f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:41:07.705678 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.705649    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5576ad-5eed-4544-9213-f5d820c5f80f-kube-api-access-nnt97" (OuterVolumeSpecName: "kube-api-access-nnt97") pod "7e5576ad-5eed-4544-9213-f5d820c5f80f" (UID: "7e5576ad-5eed-4544-9213-f5d820c5f80f"). InnerVolumeSpecName "kube-api-access-nnt97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:07.804974 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.804944    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnt97\" (UniqueName: \"kubernetes.io/projected/7e5576ad-5eed-4544-9213-f5d820c5f80f-kube-api-access-nnt97\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:07.804974 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.804971    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e5576ad-5eed-4544-9213-f5d820c5f80f-dshm\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:07.804974 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:07.804982    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5576ad-5eed-4544-9213-f5d820c5f80f-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:08.285988 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.285954    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:08.334338 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.334305    2556 generic.go:356] "Generic (PLEG): container finished" podID="7e5576ad-5eed-4544-9213-f5d820c5f80f" containerID="47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275" exitCode=0
Jul 28 01:41:08.334472 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.334347    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797" event={"ID":"7e5576ad-5eed-4544-9213-f5d820c5f80f","Type":"ContainerDied","Data":"47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275"}
Jul 28 01:41:08.334472 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.334357    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"
Jul 28 01:41:08.334472 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.334399    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797" event={"ID":"7e5576ad-5eed-4544-9213-f5d820c5f80f","Type":"ContainerDied","Data":"c30903c01be27d0dc41c4d73b751871b6c9fd38615f82fc2a22e87ed626b4621"}
Jul 28 01:41:08.334472 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.334431    2556 scope.go:117] "RemoveContainer" containerID="47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275"
Jul 28 01:41:08.343354 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.343166    2556 scope.go:117] "RemoveContainer" containerID="47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275"
Jul 28 01:41:08.343458 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:41:08.343442    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275\": container with ID starting with 47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275 not found: ID does not exist" containerID="47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275"
Jul 28 01:41:08.343502 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.343465    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275"} err="failed to get container status \"47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275\": rpc error: code = NotFound desc = could not find container \"47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275\": container with ID starting with 47e40a269e10025cb53b11da0d4ffb77b6fea89b186a1086589830f6b40aa275 not found: ID does not exist"
Jul 28 01:41:08.350588 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.350550    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"]
Jul 28 01:41:08.356789 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.356768    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8c797"]
Jul 28 01:41:08.358685 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.358658    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:08.905630 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:08.905597    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:10.285903 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:10.285845    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:10.288951 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:10.288925    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5576ad-5eed-4544-9213-f5d820c5f80f" path="/var/lib/kubelet/pods/7e5576ad-5eed-4544-9213-f5d820c5f80f/volumes"
Jul 28 01:41:15.410753 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:15.410705    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:16.521301 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:16.521267    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:17.136713 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.136681    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"]
Jul 28 01:41:17.137036 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.137024    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5576ad-5eed-4544-9213-f5d820c5f80f" containerName="main"
Jul 28 01:41:17.137082 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.137038    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5576ad-5eed-4544-9213-f5d820c5f80f" containerName="main"
Jul 28 01:41:17.137117 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.137090    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5576ad-5eed-4544-9213-f5d820c5f80f" containerName="main"
Jul 28 01:41:17.139850 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.139829    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.143872 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.143851    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 28 01:41:17.143986 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.143854    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 28 01:41:17.144459 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.144441    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-mk4mn\""
Jul 28 01:41:17.144579 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.144471    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 28 01:41:17.154951 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.154915    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"]
Jul 28 01:41:17.174758 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.174734    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:17.283848 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.283815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjfp\" (UniqueName: \"kubernetes.io/projected/bcb08aec-c073-4338-b081-7534018f20b3-kube-api-access-fhjfp\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.284019 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.283872    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.284019 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.283989    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.284135 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.284041    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb08aec-c073-4338-b081-7534018f20b3-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.284135 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.284078    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.284135 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.284103    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.384678 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.384649    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjfp\" (UniqueName: \"kubernetes.io/projected/bcb08aec-c073-4338-b081-7534018f20b3-kube-api-access-fhjfp\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.384852 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.384702    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.385009 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.384976    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.385093 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.385057    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb08aec-c073-4338-b081-7534018f20b3-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.385831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.385162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.385831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.385199    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.385831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.385370    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.385831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.385609    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.385831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.385822    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.387551 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.387493    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.387816 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.387798    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb08aec-c073-4338-b081-7534018f20b3-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.393446 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.393413    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjfp\" (UniqueName: \"kubernetes.io/projected/bcb08aec-c073-4338-b081-7534018f20b3-kube-api-access-fhjfp\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.451035 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.450999    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:17.586624 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:17.586598    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"]
Jul 28 01:41:17.589239 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:41:17.589212    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb08aec_c073_4338_b081_7534018f20b3.slice/crio-f800eaba3082c0ab90c57218649b0e26d207877cfd0d1670ed01d00517d74eee WatchSource:0}: Error finding container f800eaba3082c0ab90c57218649b0e26d207877cfd0d1670ed01d00517d74eee: Status 404 returned error can't find the container with id f800eaba3082c0ab90c57218649b0e26d207877cfd0d1670ed01d00517d74eee
Jul 28 01:41:18.381963 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:18.381928    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp" event={"ID":"bcb08aec-c073-4338-b081-7534018f20b3","Type":"ContainerStarted","Data":"ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431"}
Jul 28 01:41:18.382126 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:18.381993    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp" event={"ID":"bcb08aec-c073-4338-b081-7534018f20b3","Type":"ContainerStarted","Data":"f800eaba3082c0ab90c57218649b0e26d207877cfd0d1670ed01d00517d74eee"}
Jul 28 01:41:18.406503 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:18.401355    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp" podStartSLOduration=1.401333618 podStartE2EDuration="1.401333618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:18.39994883 +0000 UTC m=+958.572694076" watchObservedRunningTime="2026-07-28 01:41:18.401333618 +0000 UTC m=+958.574078868"
Jul 28 01:41:18.451294 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:18.451260    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:18.697703 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:18.697615    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:20.314391 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:20.314351    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:20.406818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:20.406776    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:22.256098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:22.256039    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:25.227421 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.227386    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"]
Jul 28 01:41:25.236872 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:41:25.236841    2556 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:41:25.236872 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:41:25.236872 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:41:25.236872 ip-10-0-140-37 kubenswrapper[2556]:  > containerID="ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431" cmd=["/bin/sleep","15"]
Jul 28 01:41:25.237086 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.236888    2556 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp" message="" err=<
Jul 28 01:41:25.237086 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:41:25.237086 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:41:25.237086 ip-10-0-140-37 kubenswrapper[2556]:  >
Jul 28 01:41:25.237086 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:41:25.236900    2556 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:41:25.237086 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:41:25.237086 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:41:25.237086 ip-10-0-140-37 kubenswrapper[2556]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp" podUID="bcb08aec-c073-4338-b081-7534018f20b3" containerName="main" containerID="cri-o://ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431"
Jul 28 01:41:25.237086 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.236948    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp" podUID="bcb08aec-c073-4338-b081-7534018f20b3" containerName="main" containerID="cri-o://ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431" gracePeriod=60
Jul 28 01:41:25.288043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.288007    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:25.592294 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.592274    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:25.756818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.756734    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb08aec-c073-4338-b081-7534018f20b3-tls-certs\") pod \"bcb08aec-c073-4338-b081-7534018f20b3\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") "
Jul 28 01:41:25.756818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.756779    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-dshm\") pod \"bcb08aec-c073-4338-b081-7534018f20b3\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") "
Jul 28 01:41:25.756818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.756819    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-tmp-dir\") pod \"bcb08aec-c073-4338-b081-7534018f20b3\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") "
Jul 28 01:41:25.757105 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.756925    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhjfp\" (UniqueName: \"kubernetes.io/projected/bcb08aec-c073-4338-b081-7534018f20b3-kube-api-access-fhjfp\") pod \"bcb08aec-c073-4338-b081-7534018f20b3\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") "
Jul 28 01:41:25.757105 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.757027    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-home\") pod \"bcb08aec-c073-4338-b081-7534018f20b3\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") "
Jul 28 01:41:25.757105 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.757031    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bcb08aec-c073-4338-b081-7534018f20b3" (UID: "bcb08aec-c073-4338-b081-7534018f20b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:25.757105 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.757071    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-model-cache\") pod \"bcb08aec-c073-4338-b081-7534018f20b3\" (UID: \"bcb08aec-c073-4338-b081-7534018f20b3\") "
Jul 28 01:41:25.757328 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.757285    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-home" (OuterVolumeSpecName: "home") pod "bcb08aec-c073-4338-b081-7534018f20b3" (UID: "bcb08aec-c073-4338-b081-7534018f20b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:25.757680 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.757315    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-tmp-dir\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:25.757680 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.757339    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "bcb08aec-c073-4338-b081-7534018f20b3" (UID: "bcb08aec-c073-4338-b081-7534018f20b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:25.759210 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.759185    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb08aec-c073-4338-b081-7534018f20b3-kube-api-access-fhjfp" (OuterVolumeSpecName: "kube-api-access-fhjfp") pod "bcb08aec-c073-4338-b081-7534018f20b3" (UID: "bcb08aec-c073-4338-b081-7534018f20b3"). InnerVolumeSpecName "kube-api-access-fhjfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:25.759292 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.759215    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb08aec-c073-4338-b081-7534018f20b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcb08aec-c073-4338-b081-7534018f20b3" (UID: "bcb08aec-c073-4338-b081-7534018f20b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:41:25.759292 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.759215    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-dshm" (OuterVolumeSpecName: "dshm") pod "bcb08aec-c073-4338-b081-7534018f20b3" (UID: "bcb08aec-c073-4338-b081-7534018f20b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:25.858334 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.858298    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-home\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:25.858334 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.858326    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-model-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:25.858334 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.858340    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb08aec-c073-4338-b081-7534018f20b3-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:25.858580 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.858350    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb08aec-c073-4338-b081-7534018f20b3-dshm\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:25.858580 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:25.858362    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjfp\" (UniqueName: \"kubernetes.io/projected/bcb08aec-c073-4338-b081-7534018f20b3-kube-api-access-fhjfp\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:41:26.417452 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:26.417421    2556 generic.go:356] "Generic (PLEG): container finished" podID="bcb08aec-c073-4338-b081-7534018f20b3" containerID="ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431" exitCode=0
Jul 28 01:41:26.417960 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:26.417465    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"
Jul 28 01:41:26.417960 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:26.417508    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp" event={"ID":"bcb08aec-c073-4338-b081-7534018f20b3","Type":"ContainerDied","Data":"ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431"}
Jul 28 01:41:26.417960 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:26.417599    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp" event={"ID":"bcb08aec-c073-4338-b081-7534018f20b3","Type":"ContainerDied","Data":"f800eaba3082c0ab90c57218649b0e26d207877cfd0d1670ed01d00517d74eee"}
Jul 28 01:41:26.417960 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:26.417638    2556 scope.go:117] "RemoveContainer" containerID="ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431"
Jul 28 01:41:26.426300 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:26.426284    2556 scope.go:117] "RemoveContainer" containerID="ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431"
Jul 28 01:41:26.426581 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:41:26.426540    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431\": container with ID starting with ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431 not found: ID does not exist" containerID="ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431"
Jul 28 01:41:26.426652 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:26.426583    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431"} err="failed to get container status \"ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431\": rpc error: code = NotFound desc = could not find container \"ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431\": container with ID starting with ccef85496319b65187992d2b266207873191b9e459b3f93f8b4856f1a2f14431 not found: ID does not exist"
Jul 28 01:41:26.433172 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:26.433151    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"]
Jul 28 01:41:26.435331 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:26.435310    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfl79qp"]
Jul 28 01:41:28.286581 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:28.286520    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcb08aec-c073-4338-b081-7534018f20b3" path="/var/lib/kubelet/pods/bcb08aec-c073-4338-b081-7534018f20b3/volumes"
Jul 28 01:41:30.599709 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:30.599676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:30.700342 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:30.700299    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:31.402646 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:31.402614    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:31.506694 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:31.506664    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:32.835706 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:32.835674    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:37.747295 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:37.747257    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:38.615243 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:38.615202    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:39.684229 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:39.684199    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:39.784898 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:39.784865    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:45.601956 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:45.601923    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:46.780413 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:46.780367    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:46.997297 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:46.997260    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"]
Jul 28 01:41:46.997708 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:46.997692    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb08aec-c073-4338-b081-7534018f20b3" containerName="main"
Jul 28 01:41:46.997708 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:46.997709    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb08aec-c073-4338-b081-7534018f20b3" containerName="main"
Jul 28 01:41:46.997818 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:46.997771    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb08aec-c073-4338-b081-7534018f20b3" containerName="main"
Jul 28 01:41:47.000695 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.000672    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:41:47.004050 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.004018    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-gkl9v\""
Jul 28 01:41:47.005320 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.005299    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 28 01:41:47.005462 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.005335    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 28 01:41:47.005462 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.005351    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 28 01:41:47.005462 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.005418    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-9xx4h\""
Jul 28 01:41:47.021836 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.021815    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"]
Jul 28 01:41:47.039721 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.039670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887fr\" (UniqueName: \"kubernetes.io/projected/c96476be-724c-4349-a2fa-fcc8a0b2ae85-kube-api-access-887fr\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq\" (UID: \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:41:47.039802 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.039738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96476be-724c-4349-a2fa-fcc8a0b2ae85-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq\" (UID: \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:41:47.044393 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.044373    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:47.140763 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.140736    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96476be-724c-4349-a2fa-fcc8a0b2ae85-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq\" (UID: \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:41:47.140892 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.140791    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887fr\" (UniqueName: \"kubernetes.io/projected/c96476be-724c-4349-a2fa-fcc8a0b2ae85-kube-api-access-887fr\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq\" (UID: \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:41:47.143349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.143320    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96476be-724c-4349-a2fa-fcc8a0b2ae85-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq\" (UID: \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:41:47.161710 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.161687    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-887fr\" (UniqueName: \"kubernetes.io/projected/c96476be-724c-4349-a2fa-fcc8a0b2ae85-kube-api-access-887fr\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq\" (UID: \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:41:47.310651 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.310551    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:41:47.445629 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.445591    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"]
Jul 28 01:41:47.448743 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:41:47.448715    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96476be_724c_4349_a2fa_fcc8a0b2ae85.slice/crio-184ac407625e3ad58474f21111a08b3369a5b41454d73a8fbcb9f901f213b432 WatchSource:0}: Error finding container 184ac407625e3ad58474f21111a08b3369a5b41454d73a8fbcb9f901f213b432: Status 404 returned error can't find the container with id 184ac407625e3ad58474f21111a08b3369a5b41454d73a8fbcb9f901f213b432
Jul 28 01:41:47.515227 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:47.515200    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq" event={"ID":"c96476be-724c-4349-a2fa-fcc8a0b2ae85","Type":"ContainerStarted","Data":"184ac407625e3ad58474f21111a08b3369a5b41454d73a8fbcb9f901f213b432"}
Jul 28 01:41:48.520854 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:48.520818    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq" event={"ID":"c96476be-724c-4349-a2fa-fcc8a0b2ae85","Type":"ContainerStarted","Data":"caff21c2a5e3a5d9b5e497b7a1e022b126009b9004a718fb18116b286f348844"}
Jul 28 01:41:48.521266 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:48.520936    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:41:48.539687 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:48.539628    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq" podStartSLOduration=2.539612741 podStartE2EDuration="2.539612741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:48.537682784 +0000 UTC m=+988.710428068" watchObservedRunningTime="2026-07-28 01:41:48.539612741 +0000 UTC m=+988.712357988"
Jul 28 01:41:51.080014 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:51.079980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:51.141948 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:51.141896    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:54.867930 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:54.867892    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:54.927155 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:54.927120    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:55.973820 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:55.973785    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:56.136975 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.136944    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:56.251359 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.251269    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:56.324961 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.324927    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"]
Jul 28 01:41:56.328550 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.328530    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:41:56.334606 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.334551    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 01:41:56.334769 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.334719    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-2pbh9\""
Jul 28 01:41:56.334841 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.334829    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 01:41:56.334959 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.334938    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 01:41:56.335179 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.335037    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-v5vbh\""
Jul 28 01:41:56.340806 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.340787    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"]
Jul 28 01:41:56.372510 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.372472    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:56.415406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.415369    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5fp\" (UniqueName: \"kubernetes.io/projected/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-kube-api-access-hv5fp\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9\" (UID: \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:41:56.415600 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.415425    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9\" (UID: \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:41:56.516002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.515914    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5fp\" (UniqueName: \"kubernetes.io/projected/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-kube-api-access-hv5fp\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9\" (UID: \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:41:56.516002 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.515970    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9\" (UID: \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:41:56.518649 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.518624    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9\" (UID: \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:41:56.525188 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.525168    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5fp\" (UniqueName: \"kubernetes.io/projected/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-kube-api-access-hv5fp\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9\" (UID: \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:41:56.639831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.639805    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:41:56.782349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:56.782326    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"]
Jul 28 01:41:56.784834 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:41:56.784807    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c0d1de_7895_498a_bd8b_fcc2bfde6b10.slice/crio-9f1244fefc76e4f4fad21bd5813a8df001c626e804e881d52b88e416da5700c1 WatchSource:0}: Error finding container 9f1244fefc76e4f4fad21bd5813a8df001c626e804e881d52b88e416da5700c1: Status 404 returned error can't find the container with id 9f1244fefc76e4f4fad21bd5813a8df001c626e804e881d52b88e416da5700c1
Jul 28 01:41:57.564841 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:57.564809    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9" event={"ID":"49c0d1de-7895-498a-bd8b-fcc2bfde6b10","Type":"ContainerStarted","Data":"b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c"}
Jul 28 01:41:57.565325 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:57.564882    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9" event={"ID":"49c0d1de-7895-498a-bd8b-fcc2bfde6b10","Type":"ContainerStarted","Data":"9f1244fefc76e4f4fad21bd5813a8df001c626e804e881d52b88e416da5700c1"}
Jul 28 01:41:57.565325 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:57.564917    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:41:57.582724 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:41:57.582679    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9" podStartSLOduration=1.582665223 podStartE2EDuration="1.582665223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:57.582047365 +0000 UTC m=+997.754792616" watchObservedRunningTime="2026-07-28 01:41:57.582665223 +0000 UTC m=+997.755410472"
Jul 28 01:42:02.835312 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:42:02.835280    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:42:02.937204 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:42:02.937171    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:42:08.647443 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:42:08.647410    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:42:17.950243 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:42:17.950195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:42:19.527128 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:42:19.527097    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:42:28.571062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:42:28.571032    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:44:02.945596 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:02.945537    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:44:03.032738 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:03.032709    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:44:26.493937 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:26.493902    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:44:32.096514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:32.096478    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"]
Jul 28 01:44:32.107306 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:32.107273    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:44:32.164840 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:32.164806    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:44:38.612501 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:38.612471    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"]
Jul 28 01:44:38.616972 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:38.616942    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:44:38.714699 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:38.714664    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:44:39.527199 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:39.527157    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq" podUID="c96476be-724c-4349-a2fa-fcc8a0b2ae85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:44:47.117669 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.117626    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq" podUID="c96476be-724c-4349-a2fa-fcc8a0b2ae85" containerName="main" containerID="cri-o://caff21c2a5e3a5d9b5e497b7a1e022b126009b9004a718fb18116b286f348844" gracePeriod=45
Jul 28 01:44:47.303578 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.303525    2556 generic.go:356] "Generic (PLEG): container finished" podID="c96476be-724c-4349-a2fa-fcc8a0b2ae85" containerID="caff21c2a5e3a5d9b5e497b7a1e022b126009b9004a718fb18116b286f348844" exitCode=0
Jul 28 01:44:47.303714 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.303660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq" event={"ID":"c96476be-724c-4349-a2fa-fcc8a0b2ae85","Type":"ContainerDied","Data":"caff21c2a5e3a5d9b5e497b7a1e022b126009b9004a718fb18116b286f348844"}
Jul 28 01:44:47.367096 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.367069    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:44:47.460412 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.460321    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96476be-724c-4349-a2fa-fcc8a0b2ae85-tls-certs\") pod \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\" (UID: \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\") "
Jul 28 01:44:47.460412 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.460393    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-887fr\" (UniqueName: \"kubernetes.io/projected/c96476be-724c-4349-a2fa-fcc8a0b2ae85-kube-api-access-887fr\") pod \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\" (UID: \"c96476be-724c-4349-a2fa-fcc8a0b2ae85\") "
Jul 28 01:44:47.462682 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.462618    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c96476be-724c-4349-a2fa-fcc8a0b2ae85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c96476be-724c-4349-a2fa-fcc8a0b2ae85" (UID: "c96476be-724c-4349-a2fa-fcc8a0b2ae85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:44:47.462799 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.462679    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96476be-724c-4349-a2fa-fcc8a0b2ae85-kube-api-access-887fr" (OuterVolumeSpecName: "kube-api-access-887fr") pod "c96476be-724c-4349-a2fa-fcc8a0b2ae85" (UID: "c96476be-724c-4349-a2fa-fcc8a0b2ae85"). InnerVolumeSpecName "kube-api-access-887fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:44:47.561275 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.561243    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c96476be-724c-4349-a2fa-fcc8a0b2ae85-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:44:47.561275 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:47.561270    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887fr\" (UniqueName: \"kubernetes.io/projected/c96476be-724c-4349-a2fa-fcc8a0b2ae85-kube-api-access-887fr\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:44:48.314893 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:48.314861    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq" event={"ID":"c96476be-724c-4349-a2fa-fcc8a0b2ae85","Type":"ContainerDied","Data":"184ac407625e3ad58474f21111a08b3369a5b41454d73a8fbcb9f901f213b432"}
Jul 28 01:44:48.315296 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:48.314872    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"
Jul 28 01:44:48.315296 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:48.314949    2556 scope.go:117] "RemoveContainer" containerID="caff21c2a5e3a5d9b5e497b7a1e022b126009b9004a718fb18116b286f348844"
Jul 28 01:44:48.331758 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:48.331731    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"]
Jul 28 01:44:48.337846 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:48.337820    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-hczbq"]
Jul 28 01:44:48.570791 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:48.570713    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9" podUID="49c0d1de-7895-498a-bd8b-fcc2bfde6b10" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:44:50.287675 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:50.287641    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c96476be-724c-4349-a2fa-fcc8a0b2ae85" path="/var/lib/kubelet/pods/c96476be-724c-4349-a2fa-fcc8a0b2ae85/volumes"
Jul 28 01:44:51.964420 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:51.964379    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:44:52.058131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:52.058098    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:44:53.633014 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:53.632972    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9" podUID="49c0d1de-7895-498a-bd8b-fcc2bfde6b10" containerName="main" containerID="cri-o://b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c" gracePeriod=45
Jul 28 01:44:53.908427 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:53.908403    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:44:54.016391 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.016346    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-tls-certs\") pod \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\" (UID: \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\") "
Jul 28 01:44:54.016582 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.016449    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv5fp\" (UniqueName: \"kubernetes.io/projected/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-kube-api-access-hv5fp\") pod \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\" (UID: \"49c0d1de-7895-498a-bd8b-fcc2bfde6b10\") "
Jul 28 01:44:54.018822 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.018794    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-kube-api-access-hv5fp" (OuterVolumeSpecName: "kube-api-access-hv5fp") pod "49c0d1de-7895-498a-bd8b-fcc2bfde6b10" (UID: "49c0d1de-7895-498a-bd8b-fcc2bfde6b10"). InnerVolumeSpecName "kube-api-access-hv5fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:44:54.018932 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.018879    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49c0d1de-7895-498a-bd8b-fcc2bfde6b10" (UID: "49c0d1de-7895-498a-bd8b-fcc2bfde6b10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:44:54.117081 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.117042    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5fp\" (UniqueName: \"kubernetes.io/projected/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-kube-api-access-hv5fp\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:44:54.117081 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.117067    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c0d1de-7895-498a-bd8b-fcc2bfde6b10-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:44:54.341657 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.341625    2556 generic.go:356] "Generic (PLEG): container finished" podID="49c0d1de-7895-498a-bd8b-fcc2bfde6b10" containerID="b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c" exitCode=0
Jul 28 01:44:54.341780 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.341678    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"
Jul 28 01:44:54.341780 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.341718    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9" event={"ID":"49c0d1de-7895-498a-bd8b-fcc2bfde6b10","Type":"ContainerDied","Data":"b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c"}
Jul 28 01:44:54.341780 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.341769    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9" event={"ID":"49c0d1de-7895-498a-bd8b-fcc2bfde6b10","Type":"ContainerDied","Data":"9f1244fefc76e4f4fad21bd5813a8df001c626e804e881d52b88e416da5700c1"}
Jul 28 01:44:54.341899 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.341798    2556 scope.go:117] "RemoveContainer" containerID="b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c"
Jul 28 01:44:54.350001 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.349985    2556 scope.go:117] "RemoveContainer" containerID="b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c"
Jul 28 01:44:54.350288 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:44:54.350258    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c\": container with ID starting with b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c not found: ID does not exist" containerID="b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c"
Jul 28 01:44:54.350390 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.350289    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c"} err="failed to get container status \"b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c\": rpc error: code = NotFound desc = could not find container \"b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c\": container with ID starting with b2ec627f573a0d874803d5bcecef13f9fc3d416499851aed674ef973153f2c8c not found: ID does not exist"
Jul 28 01:44:54.358642 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.358620    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"]
Jul 28 01:44:54.362236 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:54.362217    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewrvg9"]
Jul 28 01:44:56.286272 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:56.286244    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c0d1de-7895-498a-bd8b-fcc2bfde6b10" path="/var/lib/kubelet/pods/49c0d1de-7895-498a-bd8b-fcc2bfde6b10/volumes"
Jul 28 01:44:58.085964 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:44:58.085932    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:03.676254 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:03.676217    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:03.779911 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:03.779874    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:06.679689 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.679652    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:06.862685 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.862656    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"]
Jul 28 01:45:06.862993 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.862982    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c96476be-724c-4349-a2fa-fcc8a0b2ae85" containerName="main"
Jul 28 01:45:06.863049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.862995    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c96476be-724c-4349-a2fa-fcc8a0b2ae85" containerName="main"
Jul 28 01:45:06.863049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.863006    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c0d1de-7895-498a-bd8b-fcc2bfde6b10" containerName="main"
Jul 28 01:45:06.863049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.863011    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c0d1de-7895-498a-bd8b-fcc2bfde6b10" containerName="main"
Jul 28 01:45:06.863140 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.863061    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c96476be-724c-4349-a2fa-fcc8a0b2ae85" containerName="main"
Jul 28 01:45:06.863140 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.863067    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c0d1de-7895-498a-bd8b-fcc2bfde6b10" containerName="main"
Jul 28 01:45:06.866225 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.866206    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:06.871029 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.871007    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 28 01:45:06.871152 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.871091    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 28 01:45:06.871152 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.871112    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-48kwd\""
Jul 28 01:45:06.871152 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.871122    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-dm7wc\""
Jul 28 01:45:06.871321 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.871151    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 28 01:45:06.877838 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.877815    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"]
Jul 28 01:45:06.916872 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.916840    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a64d9a-87e4-49ab-9811-14363ec259ae-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp\" (UID: \"23a64d9a-87e4-49ab-9811-14363ec259ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:06.917012 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.916877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhg77\" (UniqueName: \"kubernetes.io/projected/23a64d9a-87e4-49ab-9811-14363ec259ae-kube-api-access-lhg77\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp\" (UID: \"23a64d9a-87e4-49ab-9811-14363ec259ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:06.933711 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:06.933637    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:07.018030 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.017996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a64d9a-87e4-49ab-9811-14363ec259ae-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp\" (UID: \"23a64d9a-87e4-49ab-9811-14363ec259ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:07.018171 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.018035    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhg77\" (UniqueName: \"kubernetes.io/projected/23a64d9a-87e4-49ab-9811-14363ec259ae-kube-api-access-lhg77\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp\" (UID: \"23a64d9a-87e4-49ab-9811-14363ec259ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:07.020592 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.020547    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a64d9a-87e4-49ab-9811-14363ec259ae-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp\" (UID: \"23a64d9a-87e4-49ab-9811-14363ec259ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:07.026412 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.026392    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhg77\" (UniqueName: \"kubernetes.io/projected/23a64d9a-87e4-49ab-9811-14363ec259ae-kube-api-access-lhg77\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp\" (UID: \"23a64d9a-87e4-49ab-9811-14363ec259ae\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:07.177648 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.177616    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:07.305604 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.305471    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"]
Jul 28 01:45:07.307688 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:45:07.307665    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a64d9a_87e4_49ab_9811_14363ec259ae.slice/crio-7c8d255b147271c04188f24871207669412a8c489b2b5552dac0c7442f0e156e WatchSource:0}: Error finding container 7c8d255b147271c04188f24871207669412a8c489b2b5552dac0c7442f0e156e: Status 404 returned error can't find the container with id 7c8d255b147271c04188f24871207669412a8c489b2b5552dac0c7442f0e156e
Jul 28 01:45:07.401186 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.401145    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp" event={"ID":"23a64d9a-87e4-49ab-9811-14363ec259ae","Type":"ContainerStarted","Data":"54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5"}
Jul 28 01:45:07.401337 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.401200    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp" event={"ID":"23a64d9a-87e4-49ab-9811-14363ec259ae","Type":"ContainerStarted","Data":"7c8d255b147271c04188f24871207669412a8c489b2b5552dac0c7442f0e156e"}
Jul 28 01:45:07.401337 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.401226    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:07.420922 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:07.420858    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp" podStartSLOduration=1.42083782 podStartE2EDuration="1.42083782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:45:07.418021144 +0000 UTC m=+1187.590766392" watchObservedRunningTime="2026-07-28 01:45:07.42083782 +0000 UTC m=+1187.593583071"
Jul 28 01:45:13.717757 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:13.717720    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:13.778730 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:13.778688    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:20.034116 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:20.034078    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:20.306481 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:20.306401    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:45:20.309393 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:20.309373    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:45:25.586435 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:25.586394    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:25.686888 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:25.686855    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:27.739775 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:27.739740    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:38.406958 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:38.406930    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:45:40.232026 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.231991    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"]
Jul 28 01:45:40.235059 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.235036    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.239018 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.238982    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 28 01:45:40.239150 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.238986    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-vn47b\""
Jul 28 01:45:40.239222 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.239182    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 28 01:45:40.239286 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.239250    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 28 01:45:40.245507 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.245483    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"]
Jul 28 01:45:40.277010 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.276983    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:40.410482 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.410447    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.410693 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.410495    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.410693 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.410609    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e08e3-a689-4bd9-b42d-58e8016da388-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.410832 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.410757    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.410832 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.410792    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.410832 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.410819    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv49h\" (UniqueName: \"kubernetes.io/projected/e28e08e3-a689-4bd9-b42d-58e8016da388-kube-api-access-xv49h\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.512189 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.512099    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e08e3-a689-4bd9-b42d-58e8016da388-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.512349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.512190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.512349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.512222    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.512349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.512247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv49h\" (UniqueName: \"kubernetes.io/projected/e28e08e3-a689-4bd9-b42d-58e8016da388-kube-api-access-xv49h\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.512349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.512281    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.512349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.512316    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.512702 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.512681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.512782 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.512761    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.512833 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.512788    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.514589 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.514542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.514772 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.514757    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e08e3-a689-4bd9-b42d-58e8016da388-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.522078 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.522058    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv49h\" (UniqueName: \"kubernetes.io/projected/e28e08e3-a689-4bd9-b42d-58e8016da388-kube-api-access-xv49h\") pod \"scheduler-inline-config-test-kserve-8655bc757-qbxwm\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.546743 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.546719    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:40.633622 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.633551    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:40.689403 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:40.689378    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"]
Jul 28 01:45:40.691894 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:45:40.691854    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28e08e3_a689_4bd9_b42d_58e8016da388.slice/crio-ef00f290c8081eed3956005dea6d1fd47254c94ed481d049e6d077aa63ab440d WatchSource:0}: Error finding container ef00f290c8081eed3956005dea6d1fd47254c94ed481d049e6d077aa63ab440d: Status 404 returned error can't find the container with id ef00f290c8081eed3956005dea6d1fd47254c94ed481d049e6d077aa63ab440d
Jul 28 01:45:41.562633 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:41.562598    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm" event={"ID":"e28e08e3-a689-4bd9-b42d-58e8016da388","Type":"ContainerStarted","Data":"b22351eeb1ba5227719546cf205b12961389c754bd8298ddd86e21508cabf550"}
Jul 28 01:45:41.563003 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:41.562655    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm" event={"ID":"e28e08e3-a689-4bd9-b42d-58e8016da388","Type":"ContainerStarted","Data":"ef00f290c8081eed3956005dea6d1fd47254c94ed481d049e6d077aa63ab440d"}
Jul 28 01:45:41.582912 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:41.582866    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm" podStartSLOduration=1.582851475 podStartE2EDuration="1.582851475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:45:41.58100752 +0000 UTC m=+1221.753752768" watchObservedRunningTime="2026-07-28 01:45:41.582851475 +0000 UTC m=+1221.755596724"
Jul 28 01:45:42.547254 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:42.547223    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:49.239336 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:49.239303    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:45:50.546961 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:50.546927    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:50.559475 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:50.559451    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:45:50.615091 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:45:50.615066    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:46:16.463996 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.463911    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"]
Jul 28 01:46:16.475018 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:46:16.474975    2556 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:46:16.475018 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:46:16.475018 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:46:16.475018 ip-10-0-140-37 kubenswrapper[2556]:  > containerID="b22351eeb1ba5227719546cf205b12961389c754bd8298ddd86e21508cabf550" cmd=["/bin/sleep","15"]
Jul 28 01:46:16.475326 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.475033    2556 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm" message="" err=<
Jul 28 01:46:16.475326 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:46:16.475326 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:46:16.475326 ip-10-0-140-37 kubenswrapper[2556]:  >
Jul 28 01:46:16.475326 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:46:16.475045    2556 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:46:16.475326 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:46:16.475326 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 01:46:16.475326 ip-10-0-140-37 kubenswrapper[2556]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm" podUID="e28e08e3-a689-4bd9-b42d-58e8016da388" containerName="main" containerID="cri-o://b22351eeb1ba5227719546cf205b12961389c754bd8298ddd86e21508cabf550"
Jul 28 01:46:16.475326 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.475086    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm" podUID="e28e08e3-a689-4bd9-b42d-58e8016da388" containerName="main" containerID="cri-o://b22351eeb1ba5227719546cf205b12961389c754bd8298ddd86e21508cabf550" gracePeriod=60
Jul 28 01:46:16.481342 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.481314    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:16.604218 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.604179    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm" podUID="e28e08e3-a689-4bd9-b42d-58e8016da388" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 28 01:46:16.719230 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.719148    2556 generic.go:356] "Generic (PLEG): container finished" podID="e28e08e3-a689-4bd9-b42d-58e8016da388" containerID="b22351eeb1ba5227719546cf205b12961389c754bd8298ddd86e21508cabf550" exitCode=0
Jul 28 01:46:16.719230 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.719209    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm" event={"ID":"e28e08e3-a689-4bd9-b42d-58e8016da388","Type":"ContainerDied","Data":"b22351eeb1ba5227719546cf205b12961389c754bd8298ddd86e21508cabf550"}
Jul 28 01:46:16.821287 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.821267    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:46:16.909146 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909108    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-tmp-dir\") pod \"e28e08e3-a689-4bd9-b42d-58e8016da388\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") "
Jul 28 01:46:16.909146 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909147    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-model-cache\") pod \"e28e08e3-a689-4bd9-b42d-58e8016da388\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") "
Jul 28 01:46:16.909348 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909169    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e08e3-a689-4bd9-b42d-58e8016da388-tls-certs\") pod \"e28e08e3-a689-4bd9-b42d-58e8016da388\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") "
Jul 28 01:46:16.909348 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909199    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-dshm\") pod \"e28e08e3-a689-4bd9-b42d-58e8016da388\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") "
Jul 28 01:46:16.909348 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909233    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-home\") pod \"e28e08e3-a689-4bd9-b42d-58e8016da388\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") "
Jul 28 01:46:16.909348 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909324    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv49h\" (UniqueName: \"kubernetes.io/projected/e28e08e3-a689-4bd9-b42d-58e8016da388-kube-api-access-xv49h\") pod \"e28e08e3-a689-4bd9-b42d-58e8016da388\" (UID: \"e28e08e3-a689-4bd9-b42d-58e8016da388\") "
Jul 28 01:46:16.909590 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909420    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e28e08e3-a689-4bd9-b42d-58e8016da388" (UID: "e28e08e3-a689-4bd9-b42d-58e8016da388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:46:16.909590 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909433    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-model-cache" (OuterVolumeSpecName: "model-cache") pod "e28e08e3-a689-4bd9-b42d-58e8016da388" (UID: "e28e08e3-a689-4bd9-b42d-58e8016da388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:46:16.909590 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909543    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-home" (OuterVolumeSpecName: "home") pod "e28e08e3-a689-4bd9-b42d-58e8016da388" (UID: "e28e08e3-a689-4bd9-b42d-58e8016da388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:46:16.909751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909691    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-tmp-dir\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:46:16.909751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909709    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-model-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:46:16.909751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.909724    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-home\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:46:16.911582 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.911538    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-dshm" (OuterVolumeSpecName: "dshm") pod "e28e08e3-a689-4bd9-b42d-58e8016da388" (UID: "e28e08e3-a689-4bd9-b42d-58e8016da388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:46:16.912025 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.912004    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28e08e3-a689-4bd9-b42d-58e8016da388-kube-api-access-xv49h" (OuterVolumeSpecName: "kube-api-access-xv49h") pod "e28e08e3-a689-4bd9-b42d-58e8016da388" (UID: "e28e08e3-a689-4bd9-b42d-58e8016da388"). InnerVolumeSpecName "kube-api-access-xv49h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:46:16.912104 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:16.912017    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e28e08e3-a689-4bd9-b42d-58e8016da388-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e28e08e3-a689-4bd9-b42d-58e8016da388" (UID: "e28e08e3-a689-4bd9-b42d-58e8016da388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:46:17.010200 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:17.010120    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv49h\" (UniqueName: \"kubernetes.io/projected/e28e08e3-a689-4bd9-b42d-58e8016da388-kube-api-access-xv49h\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:46:17.010200 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:17.010147    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e08e3-a689-4bd9-b42d-58e8016da388-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:46:17.010200 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:17.010158    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e08e3-a689-4bd9-b42d-58e8016da388-dshm\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:46:17.725651 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:17.725606    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm" event={"ID":"e28e08e3-a689-4bd9-b42d-58e8016da388","Type":"ContainerDied","Data":"ef00f290c8081eed3956005dea6d1fd47254c94ed481d049e6d077aa63ab440d"}
Jul 28 01:46:17.725651 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:17.725634    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"
Jul 28 01:46:17.726163 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:17.725680    2556 scope.go:117] "RemoveContainer" containerID="b22351eeb1ba5227719546cf205b12961389c754bd8298ddd86e21508cabf550"
Jul 28 01:46:17.752443 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:17.752418    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"]
Jul 28 01:46:17.758965 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:17.758944    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-qbxwm"]
Jul 28 01:46:18.287173 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:18.287143    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e28e08e3-a689-4bd9-b42d-58e8016da388" path="/var/lib/kubelet/pods/e28e08e3-a689-4bd9-b42d-58e8016da388/volumes"
Jul 28 01:46:18.429523 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:18.429490    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:18.529620 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:18.529593    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:36.510467 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:36.510420    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:36.610839 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:36.610801    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:42.550505 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:42.550472    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:47.764924 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:47.764886    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:48.090296 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.090259    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"]
Jul 28 01:46:48.091938 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.091911    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28e08e3-a689-4bd9-b42d-58e8016da388" containerName="main"
Jul 28 01:46:48.092143 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.092130    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28e08e3-a689-4bd9-b42d-58e8016da388" containerName="main"
Jul 28 01:46:48.092414 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.092376    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28e08e3-a689-4bd9-b42d-58e8016da388" containerName="main"
Jul 28 01:46:48.096509 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.096487    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:46:48.103570 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.103485    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 28 01:46:48.105347 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.104852    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 28 01:46:48.105590 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.104929    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-jf444\""
Jul 28 01:46:48.105735 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.104985    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 28 01:46:48.105863 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.105039    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-2lk4p\""
Jul 28 01:46:48.107978 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.107909    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"]
Jul 28 01:46:48.138103 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.138062    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:48.273531 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.273486    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h467s\" (UniqueName: \"kubernetes.io/projected/e24fbeb8-eebb-482f-bd0e-06856a0425fc-kube-api-access-h467s\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c\" (UID: \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:46:48.273728 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.273598    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24fbeb8-eebb-482f-bd0e-06856a0425fc-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c\" (UID: \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:46:48.374907 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.374815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24fbeb8-eebb-482f-bd0e-06856a0425fc-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c\" (UID: \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:46:48.375055 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.374905    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h467s\" (UniqueName: \"kubernetes.io/projected/e24fbeb8-eebb-482f-bd0e-06856a0425fc-kube-api-access-h467s\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c\" (UID: \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:46:48.377587 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.377545    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24fbeb8-eebb-482f-bd0e-06856a0425fc-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c\" (UID: \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:46:48.383824 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.383796    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h467s\" (UniqueName: \"kubernetes.io/projected/e24fbeb8-eebb-482f-bd0e-06856a0425fc-kube-api-access-h467s\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c\" (UID: \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:46:48.412904 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.412877    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:46:48.546665 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.546572    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"]
Jul 28 01:46:48.548824 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:46:48.548794    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24fbeb8_eebb_482f_bd0e_06856a0425fc.slice/crio-6d106fd0f5521b8a8d09e4c70e9a0a601ca097e7a24f4e64f8bccdc164eb1827 WatchSource:0}: Error finding container 6d106fd0f5521b8a8d09e4c70e9a0a601ca097e7a24f4e64f8bccdc164eb1827: Status 404 returned error can't find the container with id 6d106fd0f5521b8a8d09e4c70e9a0a601ca097e7a24f4e64f8bccdc164eb1827
Jul 28 01:46:48.819751 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.819666    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:48.869065 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.869029    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c" event={"ID":"e24fbeb8-eebb-482f-bd0e-06856a0425fc","Type":"ContainerStarted","Data":"b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4"}
Jul 28 01:46:48.869230 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.869090    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c" event={"ID":"e24fbeb8-eebb-482f-bd0e-06856a0425fc","Type":"ContainerStarted","Data":"6d106fd0f5521b8a8d09e4c70e9a0a601ca097e7a24f4e64f8bccdc164eb1827"}
Jul 28 01:46:48.869230 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.869215    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:46:48.887257 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.887214    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c" podStartSLOduration=0.88719909 podStartE2EDuration="887.19909ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:46:48.886365646 +0000 UTC m=+1289.059110915" watchObservedRunningTime="2026-07-28 01:46:48.88719909 +0000 UTC m=+1289.059944339"
Jul 28 01:46:48.894357 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:48.894334    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:57.953167 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:46:57.953130    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:08.764619 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:08.764552    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"]
Jul 28 01:47:08.765238 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:08.765209    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:08.825215 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:08.825178    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:18.407164 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:18.407126    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp" podUID="23a64d9a-87e4-49ab-9811-14363ec259ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:47:19.875362 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:19.875335    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:47:21.632147 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:21.632109    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"]
Jul 28 01:47:21.636123 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:21.636065    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:21.701229 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:21.701200    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:23.784197 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:23.784156    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp" podUID="23a64d9a-87e4-49ab-9811-14363ec259ae" containerName="main" containerID="cri-o://54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5" gracePeriod=45
Jul 28 01:47:24.021875 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.021845    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:47:24.022965 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.022943    2556 generic.go:356] "Generic (PLEG): container finished" podID="23a64d9a-87e4-49ab-9811-14363ec259ae" containerID="54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5" exitCode=0
Jul 28 01:47:24.023067 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.023015    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp" event={"ID":"23a64d9a-87e4-49ab-9811-14363ec259ae","Type":"ContainerDied","Data":"54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5"}
Jul 28 01:47:24.023119 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.023067    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp" event={"ID":"23a64d9a-87e4-49ab-9811-14363ec259ae","Type":"ContainerDied","Data":"7c8d255b147271c04188f24871207669412a8c489b2b5552dac0c7442f0e156e"}
Jul 28 01:47:24.023119 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.023088    2556 scope.go:117] "RemoveContainer" containerID="54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5"
Jul 28 01:47:24.031049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.031032    2556 scope.go:117] "RemoveContainer" containerID="54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5"
Jul 28 01:47:24.031325 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:47:24.031301    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5\": container with ID starting with 54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5 not found: ID does not exist" containerID="54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5"
Jul 28 01:47:24.031419 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.031336    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5"} err="failed to get container status \"54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5\": rpc error: code = NotFound desc = could not find container \"54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5\": container with ID starting with 54accdcad6a64e028bd2f95c8178791784ea40d27cc23a7b8694c061120379d5 not found: ID does not exist"
Jul 28 01:47:24.144252 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.144213    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a64d9a-87e4-49ab-9811-14363ec259ae-tls-certs\") pod \"23a64d9a-87e4-49ab-9811-14363ec259ae\" (UID: \"23a64d9a-87e4-49ab-9811-14363ec259ae\") "
Jul 28 01:47:24.144252 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.144252    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhg77\" (UniqueName: \"kubernetes.io/projected/23a64d9a-87e4-49ab-9811-14363ec259ae-kube-api-access-lhg77\") pod \"23a64d9a-87e4-49ab-9811-14363ec259ae\" (UID: \"23a64d9a-87e4-49ab-9811-14363ec259ae\") "
Jul 28 01:47:24.146586 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.146536    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a64d9a-87e4-49ab-9811-14363ec259ae-kube-api-access-lhg77" (OuterVolumeSpecName: "kube-api-access-lhg77") pod "23a64d9a-87e4-49ab-9811-14363ec259ae" (UID: "23a64d9a-87e4-49ab-9811-14363ec259ae"). InnerVolumeSpecName "kube-api-access-lhg77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:47:24.146702 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.146607    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a64d9a-87e4-49ab-9811-14363ec259ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23a64d9a-87e4-49ab-9811-14363ec259ae" (UID: "23a64d9a-87e4-49ab-9811-14363ec259ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:24.245092 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.245062    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a64d9a-87e4-49ab-9811-14363ec259ae-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:47:24.245092 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:24.245089    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhg77\" (UniqueName: \"kubernetes.io/projected/23a64d9a-87e4-49ab-9811-14363ec259ae-kube-api-access-lhg77\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:47:25.028497 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:25.028471    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"
Jul 28 01:47:25.044604 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:25.044581    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"]
Jul 28 01:47:25.048048 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:25.048025    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc86vdp"]
Jul 28 01:47:26.286628 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:26.286592    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23a64d9a-87e4-49ab-9811-14363ec259ae" path="/var/lib/kubelet/pods/23a64d9a-87e4-49ab-9811-14363ec259ae/volumes"
Jul 28 01:47:28.782830 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:28.782791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:28.882434 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:28.882400    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:29.875366 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:29.875327    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c" podUID="e24fbeb8-eebb-482f-bd0e-06856a0425fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:47:35.378594 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:35.378516    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:36.653914 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:36.653845    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c" podUID="e24fbeb8-eebb-482f-bd0e-06856a0425fc" containerName="main" containerID="cri-o://b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4" gracePeriod=45
Jul 28 01:47:36.901570 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:36.901531    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:47:37.051654 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.051541    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h467s\" (UniqueName: \"kubernetes.io/projected/e24fbeb8-eebb-482f-bd0e-06856a0425fc-kube-api-access-h467s\") pod \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\" (UID: \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\") "
Jul 28 01:47:37.051654 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.051648    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24fbeb8-eebb-482f-bd0e-06856a0425fc-tls-certs\") pod \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\" (UID: \"e24fbeb8-eebb-482f-bd0e-06856a0425fc\") "
Jul 28 01:47:37.053962 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.053925    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24fbeb8-eebb-482f-bd0e-06856a0425fc-kube-api-access-h467s" (OuterVolumeSpecName: "kube-api-access-h467s") pod "e24fbeb8-eebb-482f-bd0e-06856a0425fc" (UID: "e24fbeb8-eebb-482f-bd0e-06856a0425fc"). InnerVolumeSpecName "kube-api-access-h467s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:47:37.054072 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.053999    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e24fbeb8-eebb-482f-bd0e-06856a0425fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e24fbeb8-eebb-482f-bd0e-06856a0425fc" (UID: "e24fbeb8-eebb-482f-bd0e-06856a0425fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:37.081964 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.081934    2556 generic.go:356] "Generic (PLEG): container finished" podID="e24fbeb8-eebb-482f-bd0e-06856a0425fc" containerID="b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4" exitCode=0
Jul 28 01:47:37.082083 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.081980    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"
Jul 28 01:47:37.082083 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.082012    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c" event={"ID":"e24fbeb8-eebb-482f-bd0e-06856a0425fc","Type":"ContainerDied","Data":"b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4"}
Jul 28 01:47:37.082083 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.082068    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c" event={"ID":"e24fbeb8-eebb-482f-bd0e-06856a0425fc","Type":"ContainerDied","Data":"6d106fd0f5521b8a8d09e4c70e9a0a601ca097e7a24f4e64f8bccdc164eb1827"}
Jul 28 01:47:37.082193 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.082090    2556 scope.go:117] "RemoveContainer" containerID="b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4"
Jul 28 01:47:37.090603 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.090585    2556 scope.go:117] "RemoveContainer" containerID="b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4"
Jul 28 01:47:37.090860 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:47:37.090842    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4\": container with ID starting with b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4 not found: ID does not exist" containerID="b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4"
Jul 28 01:47:37.090943 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.090874    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4"} err="failed to get container status \"b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4\": rpc error: code = NotFound desc = could not find container \"b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4\": container with ID starting with b06771f6af14b2a1ecec54f786576325f900aafb1739475f98d2cbdfe92945c4 not found: ID does not exist"
Jul 28 01:47:37.102530 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.102508    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"]
Jul 28 01:47:37.104961 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.104941    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87m2c"]
Jul 28 01:47:37.152902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.152881    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h467s\" (UniqueName: \"kubernetes.io/projected/e24fbeb8-eebb-482f-bd0e-06856a0425fc-kube-api-access-h467s\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:47:37.152902 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.152903    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24fbeb8-eebb-482f-bd0e-06856a0425fc-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:47:37.976195 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:37.976145    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:38.251869 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.251793    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"]
Jul 28 01:47:38.252124 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.252112    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a64d9a-87e4-49ab-9811-14363ec259ae" containerName="main"
Jul 28 01:47:38.252167 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.252126    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a64d9a-87e4-49ab-9811-14363ec259ae" containerName="main"
Jul 28 01:47:38.252167 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.252134    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24fbeb8-eebb-482f-bd0e-06856a0425fc" containerName="main"
Jul 28 01:47:38.252167 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.252140    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24fbeb8-eebb-482f-bd0e-06856a0425fc" containerName="main"
Jul 28 01:47:38.252262 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.252192    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a64d9a-87e4-49ab-9811-14363ec259ae" containerName="main"
Jul 28 01:47:38.252262 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.252200    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24fbeb8-eebb-482f-bd0e-06856a0425fc" containerName="main"
Jul 28 01:47:38.256027 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.256006    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:47:38.260220 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.260195    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 28 01:47:38.260371 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.260302    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-7m79r\""
Jul 28 01:47:38.260445 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.260381    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-8chgm\""
Jul 28 01:47:38.260445 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.260424    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 28 01:47:38.260588 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.260546    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 28 01:47:38.270920 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.270893    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"]
Jul 28 01:47:38.286273 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.286245    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e24fbeb8-eebb-482f-bd0e-06856a0425fc" path="/var/lib/kubelet/pods/e24fbeb8-eebb-482f-bd0e-06856a0425fc/volumes"
Jul 28 01:47:38.296551 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.296521    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:38.362308 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.362273    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzk77\" (UniqueName: \"kubernetes.io/projected/bb9ea911-ad4e-4b51-a091-83f999c38bc1-kube-api-access-nzk77\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld\" (UID: \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:47:38.362440 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.362333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ea911-ad4e-4b51-a091-83f999c38bc1-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld\" (UID: \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:47:38.463275 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.463245    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzk77\" (UniqueName: \"kubernetes.io/projected/bb9ea911-ad4e-4b51-a091-83f999c38bc1-kube-api-access-nzk77\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld\" (UID: \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:47:38.463425 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.463289    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ea911-ad4e-4b51-a091-83f999c38bc1-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld\" (UID: \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:47:38.465821 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.465801    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ea911-ad4e-4b51-a091-83f999c38bc1-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld\" (UID: \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:47:38.472674 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.472651    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzk77\" (UniqueName: \"kubernetes.io/projected/bb9ea911-ad4e-4b51-a091-83f999c38bc1-kube-api-access-nzk77\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld\" (UID: \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:47:38.571789 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.571764    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:47:38.703483 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:38.703460    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"]
Jul 28 01:47:38.705623 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:47:38.705595    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9ea911_ad4e_4b51_a091_83f999c38bc1.slice/crio-e06993ec9864234659c78923055718b050f58edae481b07678a56ef40a4948ea WatchSource:0}: Error finding container e06993ec9864234659c78923055718b050f58edae481b07678a56ef40a4948ea: Status 404 returned error can't find the container with id e06993ec9864234659c78923055718b050f58edae481b07678a56ef40a4948ea
Jul 28 01:47:39.093328 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:39.093286    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld" event={"ID":"bb9ea911-ad4e-4b51-a091-83f999c38bc1","Type":"ContainerStarted","Data":"518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba"}
Jul 28 01:47:39.093843 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:39.093335    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld" event={"ID":"bb9ea911-ad4e-4b51-a091-83f999c38bc1","Type":"ContainerStarted","Data":"e06993ec9864234659c78923055718b050f58edae481b07678a56ef40a4948ea"}
Jul 28 01:47:39.093843 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:39.093366    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:47:39.113584 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:39.113511    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld" podStartSLOduration=1.113493219 podStartE2EDuration="1.113493219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:47:39.111302061 +0000 UTC m=+1339.284047307" watchObservedRunningTime="2026-07-28 01:47:39.113493219 +0000 UTC m=+1339.286238469"
Jul 28 01:47:40.843911 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:40.843863    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:40.943258 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:40.943220    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:41.517971 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:41.517892    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:41.575666 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:41.575632    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:45.144862 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:45.144831    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:46.949864 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:46.949830    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:52.401977 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:52.401948    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:52.460139 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:52.460111    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:53.064290 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:53.064259    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:53.326607 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:47:53.326570    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:02.970303 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:02.970260    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:03.068121 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:03.068086    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:10.099151 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:10.099124    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:48:20.946460 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:20.946417    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:26.280214 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:26.280180    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:26.366992 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:26.366955    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:43.602011 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:43.601972    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:43.702632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:43.702594    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:45.931315 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:45.931274    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:46.031933 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:46.031897    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:52.322769 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:52.322731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:54.019272 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:54.019221    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:54.242286 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:54.242248    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:58.418812 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:58.418776    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:58.516767 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:48:58.516734    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:20.455381 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:20.455304    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:27.556142 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:27.556105    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:27.639732 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:27.639693    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:43.188042 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:43.187999    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"]
Jul 28 01:49:43.197898 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:43.197861    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:43.270017 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:43.269980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:47.207180 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:47.207142    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:47.316694 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:47.316658    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:50.099915 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:50.099882    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld" podUID="bb9ea911-ad4e-4b51-a091-83f999c38bc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:49:53.222745 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:53.222717    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:58.208344 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.208305    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld" podUID="bb9ea911-ad4e-4b51-a091-83f999c38bc1" containerName="main" containerID="cri-o://518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba" gracePeriod=45
Jul 28 01:49:58.459887 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.459827    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:49:58.527076 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.527046    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzk77\" (UniqueName: \"kubernetes.io/projected/bb9ea911-ad4e-4b51-a091-83f999c38bc1-kube-api-access-nzk77\") pod \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\" (UID: \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\") "
Jul 28 01:49:58.527076 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.527079    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ea911-ad4e-4b51-a091-83f999c38bc1-tls-certs\") pod \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\" (UID: \"bb9ea911-ad4e-4b51-a091-83f999c38bc1\") "
Jul 28 01:49:58.529485 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.529449    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9ea911-ad4e-4b51-a091-83f999c38bc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb9ea911-ad4e-4b51-a091-83f999c38bc1" (UID: "bb9ea911-ad4e-4b51-a091-83f999c38bc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:49:58.529630 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.529579    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9ea911-ad4e-4b51-a091-83f999c38bc1-kube-api-access-nzk77" (OuterVolumeSpecName: "kube-api-access-nzk77") pod "bb9ea911-ad4e-4b51-a091-83f999c38bc1" (UID: "bb9ea911-ad4e-4b51-a091-83f999c38bc1"). InnerVolumeSpecName "kube-api-access-nzk77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:49:58.628320 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.628290    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzk77\" (UniqueName: \"kubernetes.io/projected/bb9ea911-ad4e-4b51-a091-83f999c38bc1-kube-api-access-nzk77\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:49:58.628320 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.628322    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ea911-ad4e-4b51-a091-83f999c38bc1-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:49:58.706017 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.705987    2556 generic.go:356] "Generic (PLEG): container finished" podID="bb9ea911-ad4e-4b51-a091-83f999c38bc1" containerID="518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba" exitCode=0
Jul 28 01:49:58.706192 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.706031    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"
Jul 28 01:49:58.706192 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.706034    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld" event={"ID":"bb9ea911-ad4e-4b51-a091-83f999c38bc1","Type":"ContainerDied","Data":"518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba"}
Jul 28 01:49:58.706192 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.706174    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld" event={"ID":"bb9ea911-ad4e-4b51-a091-83f999c38bc1","Type":"ContainerDied","Data":"e06993ec9864234659c78923055718b050f58edae481b07678a56ef40a4948ea"}
Jul 28 01:49:58.706301 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.706205    2556 scope.go:117] "RemoveContainer" containerID="518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba"
Jul 28 01:49:58.715167 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.715150    2556 scope.go:117] "RemoveContainer" containerID="518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba"
Jul 28 01:49:58.715426 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:49:58.715408    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba\": container with ID starting with 518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba not found: ID does not exist" containerID="518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba"
Jul 28 01:49:58.715490 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.715436    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba"} err="failed to get container status \"518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba\": rpc error: code = NotFound desc = could not find container \"518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba\": container with ID starting with 518ced42b8eb1e6790f74c0c79b8b8e223a26e44f12137a3b6638cd130d717ba not found: ID does not exist"
Jul 28 01:49:58.727079 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.727060    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"]
Jul 28 01:49:58.730514 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:58.730495    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-bl4ld"]
Jul 28 01:49:59.001043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:59.000959    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:59.103454 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:49:59.103422    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:00.288049 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:00.288018    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb9ea911-ad4e-4b51-a091-83f999c38bc1" path="/var/lib/kubelet/pods/bb9ea911-ad4e-4b51-a091-83f999c38bc1/volumes"
Jul 28 01:50:01.890979 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:01.890943    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:02.081731 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.081697    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"]
Jul 28 01:50:02.082033 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.082020    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb9ea911-ad4e-4b51-a091-83f999c38bc1" containerName="main"
Jul 28 01:50:02.082076 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.082035    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb9ea911-ad4e-4b51-a091-83f999c38bc1" containerName="main"
Jul 28 01:50:02.082115 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.082096    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb9ea911-ad4e-4b51-a091-83f999c38bc1" containerName="main"
Jul 28 01:50:02.086570 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.086533    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.091015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.090997    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 28 01:50:02.091784 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.091753    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-jsl6f\""
Jul 28 01:50:02.092692 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.092475    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-rksqq\""
Jul 28 01:50:02.092795 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.092767    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 28 01:50:02.092979 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.092963    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 28 01:50:02.095513 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.095491    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"]
Jul 28 01:50:02.124089 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.124066    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:02.161765 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.161699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9flc\" (UniqueName: \"kubernetes.io/projected/bd67e7db-240c-4380-a442-2ded66dde255-kube-api-access-l9flc\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.161765 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.161759    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd67e7db-240c-4380-a442-2ded66dde255-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.161907 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.161816    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.161907 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.161861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.262740 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.262695    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9flc\" (UniqueName: \"kubernetes.io/projected/bd67e7db-240c-4380-a442-2ded66dde255-kube-api-access-l9flc\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.262924 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.262764    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd67e7db-240c-4380-a442-2ded66dde255-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.262924 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.262784    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.262924 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.262813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.263189 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.263169    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.263265 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.263175    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.265425 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.265404    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd67e7db-240c-4380-a442-2ded66dde255-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.272159 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.272138    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9flc\" (UniqueName: \"kubernetes.io/projected/bd67e7db-240c-4380-a442-2ded66dde255-kube-api-access-l9flc\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795pm8x7\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.397518 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.397487    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.525836 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.525814    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"]
Jul 28 01:50:02.527824 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:50:02.527799    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd67e7db_240c_4380_a442_2ded66dde255.slice/crio-7cfe78e78e71f1bd5a46c4d9ea7995c70fbee4c3fa432f852a81361840063ded WatchSource:0}: Error finding container 7cfe78e78e71f1bd5a46c4d9ea7995c70fbee4c3fa432f852a81361840063ded: Status 404 returned error can't find the container with id 7cfe78e78e71f1bd5a46c4d9ea7995c70fbee4c3fa432f852a81361840063ded
Jul 28 01:50:02.726166 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.726080    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7" event={"ID":"bd67e7db-240c-4380-a442-2ded66dde255","Type":"ContainerStarted","Data":"253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707"}
Jul 28 01:50:02.726166 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.726133    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7" event={"ID":"bd67e7db-240c-4380-a442-2ded66dde255","Type":"ContainerStarted","Data":"7cfe78e78e71f1bd5a46c4d9ea7995c70fbee4c3fa432f852a81361840063ded"}
Jul 28 01:50:02.726344 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.726291    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:02.749134 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:02.749088    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7" podStartSLOduration=0.749072739 podStartE2EDuration="749.072739ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:50:02.747964705 +0000 UTC m=+1482.920709956" watchObservedRunningTime="2026-07-28 01:50:02.749072739 +0000 UTC m=+1482.921818012"
Jul 28 01:50:03.244656 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:03.244620    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:03.359718 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:03.359685    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:09.280366 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:09.280333    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:15.055378 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:15.055345    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:15.150236 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:15.150205    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:20.332426 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:20.332397    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:50:20.336545 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:20.336523    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:50:28.329754 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.329718    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"]
Jul 28 01:50:28.334724 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.334704    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.337355 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.337335    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 28 01:50:28.337491 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.337336    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 28 01:50:28.337491 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.337404    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-b8br4\""
Jul 28 01:50:28.337651 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.337604    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 28 01:50:28.343694 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.343672    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"]
Jul 28 01:50:28.363365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.363330    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:28.482127 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.482096    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgqz\" (UniqueName: \"kubernetes.io/projected/8e7a3387-a97f-4610-8872-e3ed7f6f885a-kube-api-access-9rgqz\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.482288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.482134    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.482288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.482167    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.482288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.482185    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.482288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.482200    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.482288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.482238    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.559839 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.559806    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:28.583587 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.583497    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgqz\" (UniqueName: \"kubernetes.io/projected/8e7a3387-a97f-4610-8872-e3ed7f6f885a-kube-api-access-9rgqz\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.583587 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.583537    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.583781 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.583596    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.583781 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.583622    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.583781 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.583643    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.583781 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.583686    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.583990 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.583973    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.583990 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.583984    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.584098 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.584016    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.585933 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.585916    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.586223 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.586203    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.593999 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.593976    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgqz\" (UniqueName: \"kubernetes.io/projected/8e7a3387-a97f-4610-8872-e3ed7f6f885a-kube-api-access-9rgqz\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.645744 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.645713    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:28.778277 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.778237    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"]
Jul 28 01:50:28.779984 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:50:28.779960    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7a3387_a97f_4610_8872_e3ed7f6f885a.slice/crio-25b073951878782c9e0c286f83fff4df9cdb213b6cdc1fd9c9bb2ba5d15880b6 WatchSource:0}: Error finding container 25b073951878782c9e0c286f83fff4df9cdb213b6cdc1fd9c9bb2ba5d15880b6: Status 404 returned error can't find the container with id 25b073951878782c9e0c286f83fff4df9cdb213b6cdc1fd9c9bb2ba5d15880b6
Jul 28 01:50:28.839831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.839767    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv" event={"ID":"8e7a3387-a97f-4610-8872-e3ed7f6f885a","Type":"ContainerStarted","Data":"f00b2837d4fb9c8a601ecffcc6264eaef3f8e8e95630d94300c02184ec61a534"}
Jul 28 01:50:28.839831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:28.839827    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv" event={"ID":"8e7a3387-a97f-4610-8872-e3ed7f6f885a","Type":"ContainerStarted","Data":"25b073951878782c9e0c286f83fff4df9cdb213b6cdc1fd9c9bb2ba5d15880b6"}
Jul 28 01:50:29.862032 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:29.861977    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv" podStartSLOduration=1.861957565 podStartE2EDuration="1.861957565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:50:29.861337728 +0000 UTC m=+1510.034082983" watchObservedRunningTime="2026-07-28 01:50:29.861957565 +0000 UTC m=+1510.034702813"
Jul 28 01:50:30.645779 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:30.645754    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:33.732891 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:33.732861    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:35.536726 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:35.536690    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:37.882057 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:37.882023    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"]
Jul 28 01:50:37.882499 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:37.882127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:37.958266 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:37.958234    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:38.646438 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:38.646400    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:38.658809 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:38.658788    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:38.897374 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:38.897279    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 01:50:43.732746 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:43.732699    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7" podUID="bd67e7db-240c-4380-a442-2ded66dde255" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:50:48.512635 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:48.512603    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:52.901460 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:52.901422    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7" podUID="bd67e7db-240c-4380-a442-2ded66dde255" containerName="main" containerID="cri-o://253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707" gracePeriod=45
Jul 28 01:50:53.150858 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.150828    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:53.187767 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.187704    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd67e7db-240c-4380-a442-2ded66dde255-tls-certs\") pod \"bd67e7db-240c-4380-a442-2ded66dde255\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") "
Jul 28 01:50:53.187882 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.187772    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-cachi2-cache\") pod \"bd67e7db-240c-4380-a442-2ded66dde255\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") "
Jul 28 01:50:53.187882 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.187865    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9flc\" (UniqueName: \"kubernetes.io/projected/bd67e7db-240c-4380-a442-2ded66dde255-kube-api-access-l9flc\") pod \"bd67e7db-240c-4380-a442-2ded66dde255\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") "
Jul 28 01:50:53.187962 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.187915    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-tokenizer-cache\") pod \"bd67e7db-240c-4380-a442-2ded66dde255\" (UID: \"bd67e7db-240c-4380-a442-2ded66dde255\") "
Jul 28 01:50:53.188109 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.188081    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "bd67e7db-240c-4380-a442-2ded66dde255" (UID: "bd67e7db-240c-4380-a442-2ded66dde255"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:50:53.188219 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.188198    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "bd67e7db-240c-4380-a442-2ded66dde255" (UID: "bd67e7db-240c-4380-a442-2ded66dde255"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:50:53.188278 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.188202    2556 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-cachi2-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:50:53.190154 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.190131    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd67e7db-240c-4380-a442-2ded66dde255-kube-api-access-l9flc" (OuterVolumeSpecName: "kube-api-access-l9flc") pod "bd67e7db-240c-4380-a442-2ded66dde255" (UID: "bd67e7db-240c-4380-a442-2ded66dde255"). InnerVolumeSpecName "kube-api-access-l9flc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:50:53.190255 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.190183    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd67e7db-240c-4380-a442-2ded66dde255-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd67e7db-240c-4380-a442-2ded66dde255" (UID: "bd67e7db-240c-4380-a442-2ded66dde255"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:50:53.289268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.289242    2556 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd67e7db-240c-4380-a442-2ded66dde255-tokenizer-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:50:53.289268 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.289264    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd67e7db-240c-4380-a442-2ded66dde255-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:50:53.289440 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.289276    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9flc\" (UniqueName: \"kubernetes.io/projected/bd67e7db-240c-4380-a442-2ded66dde255-kube-api-access-l9flc\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:50:53.951544 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.951509    2556 generic.go:356] "Generic (PLEG): container finished" podID="bd67e7db-240c-4380-a442-2ded66dde255" containerID="253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707" exitCode=0
Jul 28 01:50:53.951994 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.951567    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"
Jul 28 01:50:53.951994 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.951596    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7" event={"ID":"bd67e7db-240c-4380-a442-2ded66dde255","Type":"ContainerDied","Data":"253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707"}
Jul 28 01:50:53.951994 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.951655    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7" event={"ID":"bd67e7db-240c-4380-a442-2ded66dde255","Type":"ContainerDied","Data":"7cfe78e78e71f1bd5a46c4d9ea7995c70fbee4c3fa432f852a81361840063ded"}
Jul 28 01:50:53.951994 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.951687    2556 scope.go:117] "RemoveContainer" containerID="253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707"
Jul 28 01:50:53.962846 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.962806    2556 scope.go:117] "RemoveContainer" containerID="253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707"
Jul 28 01:50:53.963334 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:50:53.963312    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707\": container with ID starting with 253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707 not found: ID does not exist" containerID="253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707"
Jul 28 01:50:53.963410 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.963347    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707"} err="failed to get container status \"253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707\": rpc error: code = NotFound desc = could not find container \"253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707\": container with ID starting with 253f8e87e5822a5c0da5ddadfe885077be52d605ac46ff86df0a5c44a2874707 not found: ID does not exist"
Jul 28 01:50:53.973093 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.973072    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"]
Jul 28 01:50:53.977190 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:53.977162    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795pm8x7"]
Jul 28 01:50:54.286801 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:54.286720    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd67e7db-240c-4380-a442-2ded66dde255" path="/var/lib/kubelet/pods/bd67e7db-240c-4380-a442-2ded66dde255/volumes"
Jul 28 01:50:58.494117 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:58.494084    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:58.552336 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:50:58.552294    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:03.646168 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:03.646131    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:09.392950 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.392912    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:09.561045 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.561014    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"]
Jul 28 01:51:09.561537 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.561514    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd67e7db-240c-4380-a442-2ded66dde255" containerName="main"
Jul 28 01:51:09.561537 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.561535    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd67e7db-240c-4380-a442-2ded66dde255" containerName="main"
Jul 28 01:51:09.561749 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.561628    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd67e7db-240c-4380-a442-2ded66dde255" containerName="main"
Jul 28 01:51:09.566299 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.566279    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:09.570280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.570248    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-b874f\""
Jul 28 01:51:09.570280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.570258    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 01:51:09.570280 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.570271    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-pvjf2\""
Jul 28 01:51:09.570497 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.570350    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 01:51:09.570497 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.570372    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 01:51:09.576543 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.576523    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"]
Jul 28 01:51:09.599193 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.599167    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:09.628303 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.628276    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sk4h\" (UniqueName: \"kubernetes.io/projected/a0d4a335-cd2f-487c-bf42-5f1219b906a4-kube-api-access-5sk4h\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27\" (UID: \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:09.628418 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.628313    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d4a335-cd2f-487c-bf42-5f1219b906a4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27\" (UID: \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:09.688625 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.688522    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:09.729761 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.729734    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sk4h\" (UniqueName: \"kubernetes.io/projected/a0d4a335-cd2f-487c-bf42-5f1219b906a4-kube-api-access-5sk4h\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27\" (UID: \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:09.729877 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.729770    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d4a335-cd2f-487c-bf42-5f1219b906a4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27\" (UID: \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:09.732447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.732415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d4a335-cd2f-487c-bf42-5f1219b906a4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27\" (UID: \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:09.738108 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.738083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sk4h\" (UniqueName: \"kubernetes.io/projected/a0d4a335-cd2f-487c-bf42-5f1219b906a4-kube-api-access-5sk4h\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27\" (UID: \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:09.753955 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.753911    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:09.877384 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:09.877360    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:10.011229 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:10.011203    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"]
Jul 28 01:51:10.013860 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:51:10.013831    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d4a335_cd2f_487c_bf42_5f1219b906a4.slice/crio-bd4cbb7d4a2ab3aa9e5f6f4ba2d37593a7171fa99ebb2c66dbf6af7d52e2fac1 WatchSource:0}: Error finding container bd4cbb7d4a2ab3aa9e5f6f4ba2d37593a7171fa99ebb2c66dbf6af7d52e2fac1: Status 404 returned error can't find the container with id bd4cbb7d4a2ab3aa9e5f6f4ba2d37593a7171fa99ebb2c66dbf6af7d52e2fac1
Jul 28 01:51:10.024362 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:10.024337    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27" event={"ID":"a0d4a335-cd2f-487c-bf42-5f1219b906a4","Type":"ContainerStarted","Data":"bd4cbb7d4a2ab3aa9e5f6f4ba2d37593a7171fa99ebb2c66dbf6af7d52e2fac1"}
Jul 28 01:51:11.031358 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:11.031326    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27" event={"ID":"a0d4a335-cd2f-487c-bf42-5f1219b906a4","Type":"ContainerStarted","Data":"46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702"}
Jul 28 01:51:11.031784 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:11.031431    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:11.050114 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:11.050065    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27" podStartSLOduration=2.050049698 podStartE2EDuration="2.050049698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:51:11.048247525 +0000 UTC m=+1551.220992776" watchObservedRunningTime="2026-07-28 01:51:11.050049698 +0000 UTC m=+1551.222794946"
Jul 28 01:51:21.776923 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:21.776891    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:42.038600 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:42.038548    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:43.948981 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:43.948935    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:43.965675 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:43.965649    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"]
Jul 28 01:51:44.048774 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:44.048743    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:51:52.037973 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:52.037930    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27" podUID="a0d4a335-cd2f-487c-bf42-5f1219b906a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:51:58.985570 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:58.985488    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27" podUID="a0d4a335-cd2f-487c-bf42-5f1219b906a4" containerName="main" containerID="cri-o://46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702" gracePeriod=45
Jul 28 01:51:59.239605 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.239541    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:59.259265 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.259234    2556 generic.go:356] "Generic (PLEG): container finished" podID="a0d4a335-cd2f-487c-bf42-5f1219b906a4" containerID="46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702" exitCode=0
Jul 28 01:51:59.259400 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.259307    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27" event={"ID":"a0d4a335-cd2f-487c-bf42-5f1219b906a4","Type":"ContainerDied","Data":"46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702"}
Jul 28 01:51:59.259400 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.259342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27" event={"ID":"a0d4a335-cd2f-487c-bf42-5f1219b906a4","Type":"ContainerDied","Data":"bd4cbb7d4a2ab3aa9e5f6f4ba2d37593a7171fa99ebb2c66dbf6af7d52e2fac1"}
Jul 28 01:51:59.259400 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.259362    2556 scope.go:117] "RemoveContainer" containerID="46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702"
Jul 28 01:51:59.259537 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.259470    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"
Jul 28 01:51:59.268799 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.268775    2556 scope.go:117] "RemoveContainer" containerID="46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702"
Jul 28 01:51:59.269079 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:51:59.269059    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702\": container with ID starting with 46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702 not found: ID does not exist" containerID="46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702"
Jul 28 01:51:59.269163 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.269088    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702"} err="failed to get container status \"46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702\": rpc error: code = NotFound desc = could not find container \"46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702\": container with ID starting with 46ac346c9ee864929c052f53d2b7e96d38aa9b4695df6e59ff1f7001aa193702 not found: ID does not exist"
Jul 28 01:51:59.356462 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.356423    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sk4h\" (UniqueName: \"kubernetes.io/projected/a0d4a335-cd2f-487c-bf42-5f1219b906a4-kube-api-access-5sk4h\") pod \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\" (UID: \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\") "
Jul 28 01:51:59.356666 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.356517    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d4a335-cd2f-487c-bf42-5f1219b906a4-tls-certs\") pod \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\" (UID: \"a0d4a335-cd2f-487c-bf42-5f1219b906a4\") "
Jul 28 01:51:59.359433 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.359402    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d4a335-cd2f-487c-bf42-5f1219b906a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0d4a335-cd2f-487c-bf42-5f1219b906a4" (UID: "a0d4a335-cd2f-487c-bf42-5f1219b906a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:51:59.359632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.359410    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d4a335-cd2f-487c-bf42-5f1219b906a4-kube-api-access-5sk4h" (OuterVolumeSpecName: "kube-api-access-5sk4h") pod "a0d4a335-cd2f-487c-bf42-5f1219b906a4" (UID: "a0d4a335-cd2f-487c-bf42-5f1219b906a4"). InnerVolumeSpecName "kube-api-access-5sk4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:51:59.457508 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.457471    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sk4h\" (UniqueName: \"kubernetes.io/projected/a0d4a335-cd2f-487c-bf42-5f1219b906a4-kube-api-access-5sk4h\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:51:59.457508 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.457499    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d4a335-cd2f-487c-bf42-5f1219b906a4-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:51:59.583834 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.583797    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"]
Jul 28 01:51:59.588794 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:51:59.588768    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9wnf27"]
Jul 28 01:52:00.287887 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:00.287853    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0d4a335-cd2f-487c-bf42-5f1219b906a4" path="/var/lib/kubelet/pods/a0d4a335-cd2f-487c-bf42-5f1219b906a4/volumes"
Jul 28 01:52:03.966695 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:03.966660    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:04.063127 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:04.063094    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:10.146392 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:10.146355    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:11.317287 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.317261    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:11.505059 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.504977    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"]
Jul 28 01:52:11.505484 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.505469    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0d4a335-cd2f-487c-bf42-5f1219b906a4" containerName="main"
Jul 28 01:52:11.505541 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.505488    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0d4a335-cd2f-487c-bf42-5f1219b906a4" containerName="main"
Jul 28 01:52:11.505635 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.505622    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0d4a335-cd2f-487c-bf42-5f1219b906a4" containerName="main"
Jul 28 01:52:11.510334 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.510312    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:11.515137 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.515117    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 01:52:11.515277 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.515119    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-btl7d\""
Jul 28 01:52:11.515349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.515120    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 01:52:11.515349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.515131    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-llc24\""
Jul 28 01:52:11.515457 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.515119    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 01:52:11.525495 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.525475    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"]
Jul 28 01:52:11.549814 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.549792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:11.657093 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.657067    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtw7\" (UniqueName: \"kubernetes.io/projected/12c0f494-dda0-4776-a371-62517303273b-kube-api-access-9wtw7\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt\" (UID: \"12c0f494-dda0-4776-a371-62517303273b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:11.657213 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.657110    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c0f494-dda0-4776-a371-62517303273b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt\" (UID: \"12c0f494-dda0-4776-a371-62517303273b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:11.758234 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.758200    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtw7\" (UniqueName: \"kubernetes.io/projected/12c0f494-dda0-4776-a371-62517303273b-kube-api-access-9wtw7\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt\" (UID: \"12c0f494-dda0-4776-a371-62517303273b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:11.758384 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.758255    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c0f494-dda0-4776-a371-62517303273b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt\" (UID: \"12c0f494-dda0-4776-a371-62517303273b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:11.760901 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.760872    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c0f494-dda0-4776-a371-62517303273b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt\" (UID: \"12c0f494-dda0-4776-a371-62517303273b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:11.775754 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.775734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtw7\" (UniqueName: \"kubernetes.io/projected/12c0f494-dda0-4776-a371-62517303273b-kube-api-access-9wtw7\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt\" (UID: \"12c0f494-dda0-4776-a371-62517303273b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:11.823674 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.823650    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:11.959938 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:11.959508    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"]
Jul 28 01:52:11.961756 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:52:11.961716    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c0f494_dda0_4776_a371_62517303273b.slice/crio-bb76b79ed1be8bb472d6f2a69153c2f4dc3fdacbab382b5b607952b0d7e3d40d WatchSource:0}: Error finding container bb76b79ed1be8bb472d6f2a69153c2f4dc3fdacbab382b5b607952b0d7e3d40d: Status 404 returned error can't find the container with id bb76b79ed1be8bb472d6f2a69153c2f4dc3fdacbab382b5b607952b0d7e3d40d
Jul 28 01:52:12.318698 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:12.318617    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt" event={"ID":"12c0f494-dda0-4776-a371-62517303273b","Type":"ContainerStarted","Data":"59e8c619ae4abb4f70313e55c45a9c4ef0d34d408aec490c2fb186c4d75dae31"}
Jul 28 01:52:12.318698 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:12.318667    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt" event={"ID":"12c0f494-dda0-4776-a371-62517303273b","Type":"ContainerStarted","Data":"bb76b79ed1be8bb472d6f2a69153c2f4dc3fdacbab382b5b607952b0d7e3d40d"}
Jul 28 01:52:12.318698 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:12.318703    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:12.337226 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:12.337178    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt" podStartSLOduration=1.3371628 podStartE2EDuration="1.3371628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:52:12.335579005 +0000 UTC m=+1612.508324252" watchObservedRunningTime="2026-07-28 01:52:12.3371628 +0000 UTC m=+1612.509908049"
Jul 28 01:52:15.609251 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:15.609213    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:15.713096 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:15.713062    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:25.740465 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:25.740429    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:43.324380 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:43.324352    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:52:45.264227 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:45.264193    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"]
Jul 28 01:52:45.270241 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:45.270210    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:45.361337 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:45.361307    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:52:53.324960 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:52:53.324913    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt" podUID="12c0f494-dda0-4776-a371-62517303273b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:53:00.284481 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.284444    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt" podUID="12c0f494-dda0-4776-a371-62517303273b" containerName="main" containerID="cri-o://59e8c619ae4abb4f70313e55c45a9c4ef0d34d408aec490c2fb186c4d75dae31" gracePeriod=45
Jul 28 01:53:00.534527 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.534470    2556 generic.go:356] "Generic (PLEG): container finished" podID="12c0f494-dda0-4776-a371-62517303273b" containerID="59e8c619ae4abb4f70313e55c45a9c4ef0d34d408aec490c2fb186c4d75dae31" exitCode=0
Jul 28 01:53:00.534527 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.534525    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt" event={"ID":"12c0f494-dda0-4776-a371-62517303273b","Type":"ContainerDied","Data":"59e8c619ae4abb4f70313e55c45a9c4ef0d34d408aec490c2fb186c4d75dae31"}
Jul 28 01:53:00.534726 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.534579    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt" event={"ID":"12c0f494-dda0-4776-a371-62517303273b","Type":"ContainerDied","Data":"bb76b79ed1be8bb472d6f2a69153c2f4dc3fdacbab382b5b607952b0d7e3d40d"}
Jul 28 01:53:00.534726 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.534598    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb76b79ed1be8bb472d6f2a69153c2f4dc3fdacbab382b5b607952b0d7e3d40d"
Jul 28 01:53:00.543517 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.543492    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:53:00.659650 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.659620    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c0f494-dda0-4776-a371-62517303273b-tls-certs\") pod \"12c0f494-dda0-4776-a371-62517303273b\" (UID: \"12c0f494-dda0-4776-a371-62517303273b\") "
Jul 28 01:53:00.659831 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.659764    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wtw7\" (UniqueName: \"kubernetes.io/projected/12c0f494-dda0-4776-a371-62517303273b-kube-api-access-9wtw7\") pod \"12c0f494-dda0-4776-a371-62517303273b\" (UID: \"12c0f494-dda0-4776-a371-62517303273b\") "
Jul 28 01:53:00.662043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.662018    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c0f494-dda0-4776-a371-62517303273b-kube-api-access-9wtw7" (OuterVolumeSpecName: "kube-api-access-9wtw7") pod "12c0f494-dda0-4776-a371-62517303273b" (UID: "12c0f494-dda0-4776-a371-62517303273b"). InnerVolumeSpecName "kube-api-access-9wtw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:53:00.662163 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.662050    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c0f494-dda0-4776-a371-62517303273b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12c0f494-dda0-4776-a371-62517303273b" (UID: "12c0f494-dda0-4776-a371-62517303273b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:53:00.761208 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.761175    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c0f494-dda0-4776-a371-62517303273b-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:53:00.761208 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:00.761199    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wtw7\" (UniqueName: \"kubernetes.io/projected/12c0f494-dda0-4776-a371-62517303273b-kube-api-access-9wtw7\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:53:01.539346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:01.539315    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"
Jul 28 01:53:01.561838 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:01.561812    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"]
Jul 28 01:53:01.566891 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:01.566870    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedhfbqt"]
Jul 28 01:53:02.287277 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:02.287247    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c0f494-dda0-4776-a371-62517303273b" path="/var/lib/kubelet/pods/12c0f494-dda0-4776-a371-62517303273b/volumes"
Jul 28 01:53:04.673458 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:04.673427    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:04.775782 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:04.775732    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:10.758529 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:10.758498    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:16.498388 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:16.498355    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:16.618173 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:16.618141    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:28.983738 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:28.983705    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:29.219317 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.219287    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"]
Jul 28 01:53:29.219632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.219619    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c0f494-dda0-4776-a371-62517303273b" containerName="main"
Jul 28 01:53:29.219680 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.219633    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c0f494-dda0-4776-a371-62517303273b" containerName="main"
Jul 28 01:53:29.219716 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.219705    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c0f494-dda0-4776-a371-62517303273b" containerName="main"
Jul 28 01:53:29.222609 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.222590    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:53:29.226347 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.226326    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 01:53:29.226485 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.226390    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-7dp59\""
Jul 28 01:53:29.226485 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.226464    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 01:53:29.226485 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.226474    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 01:53:29.226699 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.226629    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-5nqpm\""
Jul 28 01:53:29.232368 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.232348    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"]
Jul 28 01:53:29.234706 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.234640    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:29.293035 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.293006    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:29.414161 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.414123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbmn8\" (UniqueName: \"kubernetes.io/projected/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-kube-api-access-hbmn8\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s\" (UID: \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:53:29.414326 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.414202    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s\" (UID: \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:53:29.514698 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.514613    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbmn8\" (UniqueName: \"kubernetes.io/projected/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-kube-api-access-hbmn8\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s\" (UID: \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:53:29.514698 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.514683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s\" (UID: \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:53:29.517459 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.517423    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s\" (UID: \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:53:29.523353 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.523330    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbmn8\" (UniqueName: \"kubernetes.io/projected/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-kube-api-access-hbmn8\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s\" (UID: \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:53:29.534065 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.534043    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:53:29.667043 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:29.667020    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"]
Jul 28 01:53:29.669443 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:53:29.669416    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f3d922_1c11_4de6_b8f9_c64ca6cfc3eb.slice/crio-22b4183989c692bf8cccf72658c984aabf74a2054c8ee1c0f4fa863ee6e9c080 WatchSource:0}: Error finding container 22b4183989c692bf8cccf72658c984aabf74a2054c8ee1c0f4fa863ee6e9c080: Status 404 returned error can't find the container with id 22b4183989c692bf8cccf72658c984aabf74a2054c8ee1c0f4fa863ee6e9c080
Jul 28 01:53:30.665546 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:30.665501    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s" event={"ID":"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb","Type":"ContainerStarted","Data":"7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826"}
Jul 28 01:53:30.665994 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:30.665587    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s" event={"ID":"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb","Type":"ContainerStarted","Data":"22b4183989c692bf8cccf72658c984aabf74a2054c8ee1c0f4fa863ee6e9c080"}
Jul 28 01:53:30.665994 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:30.665637    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:53:30.684941 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:30.684898    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s" podStartSLOduration=1.684882874 podStartE2EDuration="1.684882874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:53:30.684257102 +0000 UTC m=+1690.857002350" watchObservedRunningTime="2026-07-28 01:53:30.684882874 +0000 UTC m=+1690.857628123"
Jul 28 01:53:39.641988 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:39.641954    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:40.436473 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:40.436435    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:55.960253 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:55.960209    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:53:56.061132 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:53:56.061091    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:54:01.671183 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:01.671144    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:54:41.742301 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:41.742270    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"]
Jul 28 01:54:41.798940 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:41.798894    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:54:41.857406 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:41.857357    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:54:47.506701 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:47.506670    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:54:51.670807 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:51.670770    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s" podUID="87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:54:56.767374 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:56.767324    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s" podUID="87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb" containerName="main" containerID="cri-o://7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826" gracePeriod=45
Jul 28 01:54:57.017770 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.017706    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:54:57.059060 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.059027    2556 generic.go:356] "Generic (PLEG): container finished" podID="87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb" containerID="7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826" exitCode=0
Jul 28 01:54:57.059060 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.059061    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"
Jul 28 01:54:57.059281 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.059104    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s" event={"ID":"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb","Type":"ContainerDied","Data":"7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826"}
Jul 28 01:54:57.059281 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.059194    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s" event={"ID":"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb","Type":"ContainerDied","Data":"22b4183989c692bf8cccf72658c984aabf74a2054c8ee1c0f4fa863ee6e9c080"}
Jul 28 01:54:57.059281 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.059223    2556 scope.go:117] "RemoveContainer" containerID="7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826"
Jul 28 01:54:57.067765 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.067747    2556 scope.go:117] "RemoveContainer" containerID="7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826"
Jul 28 01:54:57.068011 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:54:57.067991    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826\": container with ID starting with 7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826 not found: ID does not exist" containerID="7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826"
Jul 28 01:54:57.068057 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.068019    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826"} err="failed to get container status \"7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826\": rpc error: code = NotFound desc = could not find container \"7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826\": container with ID starting with 7ee818c8c0caa6130162eaaf3eaeef1d1fdff0acb4693fa19b389664f9687826 not found: ID does not exist"
Jul 28 01:54:57.160769 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.160744    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbmn8\" (UniqueName: \"kubernetes.io/projected/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-kube-api-access-hbmn8\") pod \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\" (UID: \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\") "
Jul 28 01:54:57.160922 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.160835    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-tls-certs\") pod \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\" (UID: \"87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb\") "
Jul 28 01:54:57.163030 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.163006    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-kube-api-access-hbmn8" (OuterVolumeSpecName: "kube-api-access-hbmn8") pod "87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb" (UID: "87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb"). InnerVolumeSpecName "kube-api-access-hbmn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:54:57.163149 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.163039    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb" (UID: "87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:54:57.261767 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.261742    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbmn8\" (UniqueName: \"kubernetes.io/projected/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-kube-api-access-hbmn8\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:54:57.261767 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.261765    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:54:57.381085 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.381057    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"]
Jul 28 01:54:57.386777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:57.386754    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65thm7s"]
Jul 28 01:54:58.286110 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:54:58.286077    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb" path="/var/lib/kubelet/pods/87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb/volumes"
Jul 28 01:55:01.721157 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:01.721127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:01.817024 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:01.816992    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:07.852157 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:07.852108    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:09.784925 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:09.784890    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:10.066382 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.066301    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"]
Jul 28 01:55:10.066729 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.066714    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb" containerName="main"
Jul 28 01:55:10.066778 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.066731    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb" containerName="main"
Jul 28 01:55:10.066811 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.066797    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="87f3d922-1c11-4de6-b8f9-c64ca6cfc3eb" containerName="main"
Jul 28 01:55:10.071130 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.071108    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:10.071999 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.071970    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:10.075971 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.075950    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 01:55:10.076104 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.075973    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 01:55:10.076104 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.075953    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-kqnzw\""
Jul 28 01:55:10.076262 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.076242    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-7dbtl\""
Jul 28 01:55:10.076314 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.076247    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 01:55:10.083610 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.083587    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"]
Jul 28 01:55:10.139415 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.139381    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:10.170969 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.170943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496629bd-5503-4705-9c79-7177b69b3035-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5\" (UID: \"496629bd-5503-4705-9c79-7177b69b3035\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:10.171075 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.170975    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7z9r\" (UniqueName: \"kubernetes.io/projected/496629bd-5503-4705-9c79-7177b69b3035-kube-api-access-w7z9r\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5\" (UID: \"496629bd-5503-4705-9c79-7177b69b3035\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:10.272142 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.272111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496629bd-5503-4705-9c79-7177b69b3035-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5\" (UID: \"496629bd-5503-4705-9c79-7177b69b3035\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:10.272358 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.272152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7z9r\" (UniqueName: \"kubernetes.io/projected/496629bd-5503-4705-9c79-7177b69b3035-kube-api-access-w7z9r\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5\" (UID: \"496629bd-5503-4705-9c79-7177b69b3035\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:10.274916 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.274890    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496629bd-5503-4705-9c79-7177b69b3035-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5\" (UID: \"496629bd-5503-4705-9c79-7177b69b3035\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:10.280126 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.280099    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7z9r\" (UniqueName: \"kubernetes.io/projected/496629bd-5503-4705-9c79-7177b69b3035-kube-api-access-w7z9r\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5\" (UID: \"496629bd-5503-4705-9c79-7177b69b3035\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:10.381777 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.381749    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:10.511311 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:10.511094    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"]
Jul 28 01:55:10.513591 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:55:10.513548    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496629bd_5503_4705_9c79_7177b69b3035.slice/crio-062577ffd777ab37226b64a92c30896e54e629d5ef9f0ba662564b5a7a891a1c WatchSource:0}: Error finding container 062577ffd777ab37226b64a92c30896e54e629d5ef9f0ba662564b5a7a891a1c: Status 404 returned error can't find the container with id 062577ffd777ab37226b64a92c30896e54e629d5ef9f0ba662564b5a7a891a1c
Jul 28 01:55:11.129133 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:11.129104    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5" event={"ID":"496629bd-5503-4705-9c79-7177b69b3035","Type":"ContainerStarted","Data":"48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573"}
Jul 28 01:55:11.129491 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:11.129155    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5" event={"ID":"496629bd-5503-4705-9c79-7177b69b3035","Type":"ContainerStarted","Data":"062577ffd777ab37226b64a92c30896e54e629d5ef9f0ba662564b5a7a891a1c"}
Jul 28 01:55:11.129491 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:11.129182    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:11.148202 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:11.148154    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5" podStartSLOduration=1.148136546 podStartE2EDuration="1.148136546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:55:11.147166907 +0000 UTC m=+1791.319912156" watchObservedRunningTime="2026-07-28 01:55:11.148136546 +0000 UTC m=+1791.320881798"
Jul 28 01:55:13.540126 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:13.540089    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:13.609367 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:13.609336    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:20.366328 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:20.366298    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:55:20.367954 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:20.367934    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 01:55:21.876969 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:21.876933    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:31.165681 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:31.165645    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:42.134752 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:42.134719    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:55:48.768968 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:48.768924    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:48.780766 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:48.780736    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"]
Jul 28 01:55:48.836518 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:48.836486    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:55:52.135020 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:55:52.134981    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5" podUID="496629bd-5503-4705-9c79-7177b69b3035" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:56:02.134240 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:02.134204    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5" podUID="496629bd-5503-4705-9c79-7177b69b3035" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:56:03.800652 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:03.800616    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5" podUID="496629bd-5503-4705-9c79-7177b69b3035" containerName="main" containerID="cri-o://48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573" gracePeriod=45
Jul 28 01:56:04.052797 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.052742    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:56:04.228716 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.228687    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496629bd-5503-4705-9c79-7177b69b3035-tls-certs\") pod \"496629bd-5503-4705-9c79-7177b69b3035\" (UID: \"496629bd-5503-4705-9c79-7177b69b3035\") "
Jul 28 01:56:04.228901 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.228785    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7z9r\" (UniqueName: \"kubernetes.io/projected/496629bd-5503-4705-9c79-7177b69b3035-kube-api-access-w7z9r\") pod \"496629bd-5503-4705-9c79-7177b69b3035\" (UID: \"496629bd-5503-4705-9c79-7177b69b3035\") "
Jul 28 01:56:04.231137 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.231107    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496629bd-5503-4705-9c79-7177b69b3035-kube-api-access-w7z9r" (OuterVolumeSpecName: "kube-api-access-w7z9r") pod "496629bd-5503-4705-9c79-7177b69b3035" (UID: "496629bd-5503-4705-9c79-7177b69b3035"). InnerVolumeSpecName "kube-api-access-w7z9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:56:04.231137 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.231121    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496629bd-5503-4705-9c79-7177b69b3035-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "496629bd-5503-4705-9c79-7177b69b3035" (UID: "496629bd-5503-4705-9c79-7177b69b3035"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:56:04.329689 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.329668    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496629bd-5503-4705-9c79-7177b69b3035-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:56:04.329689 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.329692    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7z9r\" (UniqueName: \"kubernetes.io/projected/496629bd-5503-4705-9c79-7177b69b3035-kube-api-access-w7z9r\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:56:04.362163 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.362134    2556 generic.go:356] "Generic (PLEG): container finished" podID="496629bd-5503-4705-9c79-7177b69b3035" containerID="48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573" exitCode=0
Jul 28 01:56:04.362291 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.362184    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"
Jul 28 01:56:04.362291 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.362221    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5" event={"ID":"496629bd-5503-4705-9c79-7177b69b3035","Type":"ContainerDied","Data":"48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573"}
Jul 28 01:56:04.362291 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.362269    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5" event={"ID":"496629bd-5503-4705-9c79-7177b69b3035","Type":"ContainerDied","Data":"062577ffd777ab37226b64a92c30896e54e629d5ef9f0ba662564b5a7a891a1c"}
Jul 28 01:56:04.362291 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.362292    2556 scope.go:117] "RemoveContainer" containerID="48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573"
Jul 28 01:56:04.372523 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.372492    2556 scope.go:117] "RemoveContainer" containerID="48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573"
Jul 28 01:56:04.372914 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:56:04.372890    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573\": container with ID starting with 48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573 not found: ID does not exist" containerID="48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573"
Jul 28 01:56:04.372971 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.372923    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573"} err="failed to get container status \"48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573\": rpc error: code = NotFound desc = could not find container \"48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573\": container with ID starting with 48f6dfbe7d4d66cb6e87288c6291d02db06e8967b9ce312fc3c917bc7e480573 not found: ID does not exist"
Jul 28 01:56:04.380794 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.380772    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"]
Jul 28 01:56:04.384540 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:04.384521    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9r2zc5"]
Jul 28 01:56:06.287340 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:06.287305    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="496629bd-5503-4705-9c79-7177b69b3035" path="/var/lib/kubelet/pods/496629bd-5503-4705-9c79-7177b69b3035/volumes"
Jul 28 01:56:08.578110 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:08.578079    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:56:08.669611 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:08.669581    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:56:14.877904 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:14.877871    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:56:20.088807 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.088774    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:56:20.314917 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.314886    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"]
Jul 28 01:56:20.315223 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.315210    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496629bd-5503-4705-9c79-7177b69b3035" containerName="main"
Jul 28 01:56:20.315223 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.315223    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="496629bd-5503-4705-9c79-7177b69b3035" containerName="main"
Jul 28 01:56:20.315350 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.315288    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="496629bd-5503-4705-9c79-7177b69b3035" containerName="main"
Jul 28 01:56:20.319701 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.319681    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:20.324314 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.324053    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 28 01:56:20.324314 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.324072    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-gp4d9\""
Jul 28 01:56:20.324314 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.324127    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 28 01:56:20.324314 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.324072    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-25679\""
Jul 28 01:56:20.324314 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.324130    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 28 01:56:20.330389 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.330368    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"]
Jul 28 01:56:20.353746 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.353677    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:56:20.361801 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.361781    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw\" (UID: \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:20.361888 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.361863    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkwr\" (UniqueName: \"kubernetes.io/projected/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-kube-api-access-2bkwr\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw\" (UID: \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:20.462365 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.462334    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkwr\" (UniqueName: \"kubernetes.io/projected/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-kube-api-access-2bkwr\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw\" (UID: \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:20.462511 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.462399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw\" (UID: \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:20.465110 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.465079    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw\" (UID: \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:20.472592 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.472541    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkwr\" (UniqueName: \"kubernetes.io/projected/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-kube-api-access-2bkwr\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw\" (UID: \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:20.631247 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.631164    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:20.766910 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:20.766695    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"]
Jul 28 01:56:20.769075 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:56:20.769037    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfa917a_4b05_44b1_9f90_51f24a64f9ba.slice/crio-924ef795634cc619e395f3b4158ff2e9f397234a480183aa7a0510d057891142 WatchSource:0}: Error finding container 924ef795634cc619e395f3b4158ff2e9f397234a480183aa7a0510d057891142: Status 404 returned error can't find the container with id 924ef795634cc619e395f3b4158ff2e9f397234a480183aa7a0510d057891142
Jul 28 01:56:21.442035 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:21.442000    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw" event={"ID":"8dfa917a-4b05-44b1-9f90-51f24a64f9ba","Type":"ContainerStarted","Data":"2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40"}
Jul 28 01:56:21.442453 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:21.442054    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw" event={"ID":"8dfa917a-4b05-44b1-9f90-51f24a64f9ba","Type":"ContainerStarted","Data":"924ef795634cc619e395f3b4158ff2e9f397234a480183aa7a0510d057891142"}
Jul 28 01:56:21.442453 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:21.442078    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:21.460680 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:21.460610    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw" podStartSLOduration=1.460595295 podStartE2EDuration="1.460595295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:56:21.458809772 +0000 UTC m=+1861.631555022" watchObservedRunningTime="2026-07-28 01:56:21.460595295 +0000 UTC m=+1861.633340544"
Jul 28 01:56:21.641856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:21.641822    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:56:21.742436 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:21.742350    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:56:32.355363 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:32.355329    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:56:52.447850 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:52.447821    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:56:53.961703 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:53.961670    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:56:53.968420 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:53.968392    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"]
Jul 28 01:56:54.071113 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:56:54.071070    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:02.447408 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:02.447364    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw" podUID="8dfa917a-4b05-44b1-9f90-51f24a64f9ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:57:08.988469 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:08.988423    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw" podUID="8dfa917a-4b05-44b1-9f90-51f24a64f9ba" containerName="main" containerID="cri-o://2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40" gracePeriod=45
Jul 28 01:57:09.244071 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.244010    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:57:09.272797 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.272749    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-tls-certs\") pod \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\" (UID: \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\") "
Jul 28 01:57:09.272961 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.272854    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bkwr\" (UniqueName: \"kubernetes.io/projected/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-kube-api-access-2bkwr\") pod \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\" (UID: \"8dfa917a-4b05-44b1-9f90-51f24a64f9ba\") "
Jul 28 01:57:09.275204 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.275171    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-kube-api-access-2bkwr" (OuterVolumeSpecName: "kube-api-access-2bkwr") pod "8dfa917a-4b05-44b1-9f90-51f24a64f9ba" (UID: "8dfa917a-4b05-44b1-9f90-51f24a64f9ba"). InnerVolumeSpecName "kube-api-access-2bkwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:57:09.275204 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.275187    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dfa917a-4b05-44b1-9f90-51f24a64f9ba" (UID: "8dfa917a-4b05-44b1-9f90-51f24a64f9ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:57:09.373475 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.373448    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:57:09.373475 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.373476    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bkwr\" (UniqueName: \"kubernetes.io/projected/8dfa917a-4b05-44b1-9f90-51f24a64f9ba-kube-api-access-2bkwr\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:57:09.664891 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.664861    2556 generic.go:356] "Generic (PLEG): container finished" podID="8dfa917a-4b05-44b1-9f90-51f24a64f9ba" containerID="2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40" exitCode=0
Jul 28 01:57:09.665062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.664898    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw" event={"ID":"8dfa917a-4b05-44b1-9f90-51f24a64f9ba","Type":"ContainerDied","Data":"2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40"}
Jul 28 01:57:09.665062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.664907    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"
Jul 28 01:57:09.665062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.664946    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw" event={"ID":"8dfa917a-4b05-44b1-9f90-51f24a64f9ba","Type":"ContainerDied","Data":"924ef795634cc619e395f3b4158ff2e9f397234a480183aa7a0510d057891142"}
Jul 28 01:57:09.665062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.664971    2556 scope.go:117] "RemoveContainer" containerID="2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40"
Jul 28 01:57:09.673870 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.673854    2556 scope.go:117] "RemoveContainer" containerID="2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40"
Jul 28 01:57:09.674104 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:57:09.674087    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40\": container with ID starting with 2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40 not found: ID does not exist" containerID="2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40"
Jul 28 01:57:09.674163 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.674117    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40"} err="failed to get container status \"2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40\": rpc error: code = NotFound desc = could not find container \"2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40\": container with ID starting with 2319a5405f6263b1987ff22a669492121130f16908630250e3bfd3049c436c40 not found: ID does not exist"
Jul 28 01:57:09.686998 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.686975    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"]
Jul 28 01:57:09.692820 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:09.692797    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb2kcw"]
Jul 28 01:57:10.286830 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:10.286796    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dfa917a-4b05-44b1-9f90-51f24a64f9ba" path="/var/lib/kubelet/pods/8dfa917a-4b05-44b1-9f90-51f24a64f9ba/volumes"
Jul 28 01:57:13.673336 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:13.673293    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:13.764027 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:13.763984    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:19.798342 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:19.798301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:24.272196 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.272160    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:24.510883 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.510850    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"]
Jul 28 01:57:24.511219 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.511201    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfa917a-4b05-44b1-9f90-51f24a64f9ba" containerName="main"
Jul 28 01:57:24.511315 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.511223    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfa917a-4b05-44b1-9f90-51f24a64f9ba" containerName="main"
Jul 28 01:57:24.511362 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.511338    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dfa917a-4b05-44b1-9f90-51f24a64f9ba" containerName="main"
Jul 28 01:57:24.514447 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.514428    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:57:24.518230 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.518203    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 01:57:24.518359 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.518205    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-6js2f\""
Jul 28 01:57:24.518359 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.518258    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 01:57:24.518359 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.518215    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-9z8fz\""
Jul 28 01:57:24.518496 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.518360    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 28 01:57:24.523663 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.523613    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"]
Jul 28 01:57:24.531965 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.531944    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:24.593012 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.592988    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vql9p\" (UniqueName: \"kubernetes.io/projected/b1db73cc-972b-418d-855c-b9d031b5af06-kube-api-access-vql9p\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9\" (UID: \"b1db73cc-972b-418d-855c-b9d031b5af06\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:57:24.593124 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.593019    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1db73cc-972b-418d-855c-b9d031b5af06-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9\" (UID: \"b1db73cc-972b-418d-855c-b9d031b5af06\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:57:24.694116 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.694083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vql9p\" (UniqueName: \"kubernetes.io/projected/b1db73cc-972b-418d-855c-b9d031b5af06-kube-api-access-vql9p\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9\" (UID: \"b1db73cc-972b-418d-855c-b9d031b5af06\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:57:24.694116 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.694117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1db73cc-972b-418d-855c-b9d031b5af06-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9\" (UID: \"b1db73cc-972b-418d-855c-b9d031b5af06\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:57:24.696720 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.696703    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1db73cc-972b-418d-855c-b9d031b5af06-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9\" (UID: \"b1db73cc-972b-418d-855c-b9d031b5af06\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:57:24.727655 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.727625    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vql9p\" (UniqueName: \"kubernetes.io/projected/b1db73cc-972b-418d-855c-b9d031b5af06-kube-api-access-vql9p\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9\" (UID: \"b1db73cc-972b-418d-855c-b9d031b5af06\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:57:24.824762 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.824738    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:57:24.958233 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:24.958213    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"]
Jul 28 01:57:24.960653 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:57:24.960626    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1db73cc_972b_418d_855c_b9d031b5af06.slice/crio-f6ce8fcff64714510337110dcf263ff6e0145984b89b96264582422c0d716713 WatchSource:0}: Error finding container f6ce8fcff64714510337110dcf263ff6e0145984b89b96264582422c0d716713: Status 404 returned error can't find the container with id f6ce8fcff64714510337110dcf263ff6e0145984b89b96264582422c0d716713
Jul 28 01:57:25.741677 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:25.741642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9" event={"ID":"b1db73cc-972b-418d-855c-b9d031b5af06","Type":"ContainerStarted","Data":"008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62"}
Jul 28 01:57:25.742154 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:25.741706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9" event={"ID":"b1db73cc-972b-418d-855c-b9d031b5af06","Type":"ContainerStarted","Data":"f6ce8fcff64714510337110dcf263ff6e0145984b89b96264582422c0d716713"}
Jul 28 01:57:25.742154 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:25.741777    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:57:25.759137 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:25.759087    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9" podStartSLOduration=1.759069687 podStartE2EDuration="1.759069687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:57:25.758815158 +0000 UTC m=+1925.931560408" watchObservedRunningTime="2026-07-28 01:57:25.759069687 +0000 UTC m=+1925.931814937"
Jul 28 01:57:26.760316 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:26.760281    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:26.861632 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:26.861600    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:39.237213 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:39.237170    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:56.748387 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:57:56.748358    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:58:11.829229 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:11.829138    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:12.067894 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.067864    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"]
Jul 28 01:58:12.071295 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.071279    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:12.075584 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.075545    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 28 01:58:12.075693 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.075614    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-cxcfh\""
Jul 28 01:58:12.081229 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.081177    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"]
Jul 28 01:58:12.082132 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.082114    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhd5\" (UniqueName: \"kubernetes.io/projected/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-kube-api-access-glhd5\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt\" (UID: \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:12.082213 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.082156    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt\" (UID: \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:12.092649 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.092618    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:12.182906 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.182880    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhd5\" (UniqueName: \"kubernetes.io/projected/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-kube-api-access-glhd5\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt\" (UID: \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:12.183062 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.182934    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt\" (UID: \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:12.185533 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.185514    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt\" (UID: \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:12.192277 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.192255    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhd5\" (UniqueName: \"kubernetes.io/projected/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-kube-api-access-glhd5\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt\" (UID: \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:12.382705 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.382631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:12.516382 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.516356    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"]
Jul 28 01:58:12.519270 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:58:12.519230    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf76b67_7df3_4c19_8a99_e6396eac1cf0.slice/crio-b238e7c6dbd19f5774f95699a99f0fd135c6247a2432c1d140f16a4e2069539e WatchSource:0}: Error finding container b238e7c6dbd19f5774f95699a99f0fd135c6247a2432c1d140f16a4e2069539e: Status 404 returned error can't find the container with id b238e7c6dbd19f5774f95699a99f0fd135c6247a2432c1d140f16a4e2069539e
Jul 28 01:58:12.957999 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.957959    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt" event={"ID":"1bf76b67-7df3-4c19-8a99-e6396eac1cf0","Type":"ContainerStarted","Data":"1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba"}
Jul 28 01:58:12.958422 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.958052    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt" event={"ID":"1bf76b67-7df3-4c19-8a99-e6396eac1cf0","Type":"ContainerStarted","Data":"b238e7c6dbd19f5774f95699a99f0fd135c6247a2432c1d140f16a4e2069539e"}
Jul 28 01:58:12.958422 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.958090    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:12.976821 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:12.976778    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt" podStartSLOduration=0.976764895 podStartE2EDuration="976.764895ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:58:12.975144905 +0000 UTC m=+1973.147890180" watchObservedRunningTime="2026-07-28 01:58:12.976764895 +0000 UTC m=+1973.149510144"
Jul 28 01:58:20.482288 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:20.482256    2556 scope.go:117] "RemoveContainer" containerID="59e8c619ae4abb4f70313e55c45a9c4ef0d34d408aec490c2fb186c4d75dae31"
Jul 28 01:58:27.865857 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:27.865818    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:43.964245 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:43.964215    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:58:45.260299 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:45.260265    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"]
Jul 28 01:58:45.294203 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:45.294159    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:45.357464 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:45.357428    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:53.964427 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:58:53.964386    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt" podUID="1bf76b67-7df3-4c19-8a99-e6396eac1cf0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:59:00.280709 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:00.280662    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt" podUID="1bf76b67-7df3-4c19-8a99-e6396eac1cf0" containerName="main" containerID="cri-o://1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba" gracePeriod=45
Jul 28 01:59:00.525015 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:00.524986    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:59:00.682472 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:00.682439    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-tls-certs\") pod \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\" (UID: \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\") "
Jul 28 01:59:00.682659 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:00.682493    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glhd5\" (UniqueName: \"kubernetes.io/projected/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-kube-api-access-glhd5\") pod \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\" (UID: \"1bf76b67-7df3-4c19-8a99-e6396eac1cf0\") "
Jul 28 01:59:00.684883 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:00.684852    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bf76b67-7df3-4c19-8a99-e6396eac1cf0" (UID: "1bf76b67-7df3-4c19-8a99-e6396eac1cf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:59:00.685000 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:00.684898    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-kube-api-access-glhd5" (OuterVolumeSpecName: "kube-api-access-glhd5") pod "1bf76b67-7df3-4c19-8a99-e6396eac1cf0" (UID: "1bf76b67-7df3-4c19-8a99-e6396eac1cf0"). InnerVolumeSpecName "kube-api-access-glhd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:59:00.783067 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:00.783040    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:59:00.783067 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:00.783070    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhd5\" (UniqueName: \"kubernetes.io/projected/1bf76b67-7df3-4c19-8a99-e6396eac1cf0-kube-api-access-glhd5\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:59:01.174484 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:01.174449    2556 generic.go:356] "Generic (PLEG): container finished" podID="1bf76b67-7df3-4c19-8a99-e6396eac1cf0" containerID="1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba" exitCode=0
Jul 28 01:59:01.174681 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:01.174499    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"
Jul 28 01:59:01.174681 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:01.174518    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt" event={"ID":"1bf76b67-7df3-4c19-8a99-e6396eac1cf0","Type":"ContainerDied","Data":"1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba"}
Jul 28 01:59:01.174681 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:01.174576    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt" event={"ID":"1bf76b67-7df3-4c19-8a99-e6396eac1cf0","Type":"ContainerDied","Data":"b238e7c6dbd19f5774f95699a99f0fd135c6247a2432c1d140f16a4e2069539e"}
Jul 28 01:59:01.174681 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:01.174602    2556 scope.go:117] "RemoveContainer" containerID="1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba"
Jul 28 01:59:01.186349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:01.185262    2556 scope.go:117] "RemoveContainer" containerID="1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba"
Jul 28 01:59:01.186349 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:59:01.185620    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba\": container with ID starting with 1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba not found: ID does not exist" containerID="1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba"
Jul 28 01:59:01.186349 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:01.185652    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba"} err="failed to get container status \"1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba\": rpc error: code = NotFound desc = could not find container \"1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba\": container with ID starting with 1829b1b496e6aae2f3a4f86f68e394d7cf2e2a5a906c35b78be7afbed37302ba not found: ID does not exist"
Jul 28 01:59:01.197220 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:01.197196    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"]
Jul 28 01:59:01.199763 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:01.199742    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef4ckt"]
Jul 28 01:59:02.286451 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:02.286414    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf76b67-7df3-4c19-8a99-e6396eac1cf0" path="/var/lib/kubelet/pods/1bf76b67-7df3-4c19-8a99-e6396eac1cf0/volumes"
Jul 28 01:59:05.436304 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:05.436266    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"]
Jul 28 01:59:05.438122 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:05.438059    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:05.530882 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:05.530848    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:06.748105 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:06.748063    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9" podUID="b1db73cc-972b-418d-855c-b9d031b5af06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:59:16.748346 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:16.748305    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9" podUID="b1db73cc-972b-418d-855c-b9d031b5af06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:59:20.455669 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:20.455616    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9" podUID="b1db73cc-972b-418d-855c-b9d031b5af06" containerName="main" containerID="cri-o://008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62" gracePeriod=45
Jul 28 01:59:20.705159 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:20.705135    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:59:20.743739 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:20.743678    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vql9p\" (UniqueName: \"kubernetes.io/projected/b1db73cc-972b-418d-855c-b9d031b5af06-kube-api-access-vql9p\") pod \"b1db73cc-972b-418d-855c-b9d031b5af06\" (UID: \"b1db73cc-972b-418d-855c-b9d031b5af06\") "
Jul 28 01:59:20.743856 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:20.743742    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1db73cc-972b-418d-855c-b9d031b5af06-tls-certs\") pod \"b1db73cc-972b-418d-855c-b9d031b5af06\" (UID: \"b1db73cc-972b-418d-855c-b9d031b5af06\") "
Jul 28 01:59:20.745984 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:20.745951    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1db73cc-972b-418d-855c-b9d031b5af06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1db73cc-972b-418d-855c-b9d031b5af06" (UID: "b1db73cc-972b-418d-855c-b9d031b5af06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:59:20.745984 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:20.745966    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1db73cc-972b-418d-855c-b9d031b5af06-kube-api-access-vql9p" (OuterVolumeSpecName: "kube-api-access-vql9p") pod "b1db73cc-972b-418d-855c-b9d031b5af06" (UID: "b1db73cc-972b-418d-855c-b9d031b5af06"). InnerVolumeSpecName "kube-api-access-vql9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:59:20.844968 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:20.844928    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1db73cc-972b-418d-855c-b9d031b5af06-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:59:20.844968 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:20.844964    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vql9p\" (UniqueName: \"kubernetes.io/projected/b1db73cc-972b-418d-855c-b9d031b5af06-kube-api-access-vql9p\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 01:59:21.272314 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:21.272282    2556 generic.go:356] "Generic (PLEG): container finished" podID="b1db73cc-972b-418d-855c-b9d031b5af06" containerID="008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62" exitCode=0
Jul 28 01:59:21.272478 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:21.272328    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"
Jul 28 01:59:21.272478 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:21.272328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9" event={"ID":"b1db73cc-972b-418d-855c-b9d031b5af06","Type":"ContainerDied","Data":"008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62"}
Jul 28 01:59:21.272478 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:21.272456    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9" event={"ID":"b1db73cc-972b-418d-855c-b9d031b5af06","Type":"ContainerDied","Data":"f6ce8fcff64714510337110dcf263ff6e0145984b89b96264582422c0d716713"}
Jul 28 01:59:21.272662 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:21.272490    2556 scope.go:117] "RemoveContainer" containerID="008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62"
Jul 28 01:59:21.280726 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:21.280704    2556 scope.go:117] "RemoveContainer" containerID="008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62"
Jul 28 01:59:21.280970 ip-10-0-140-37 kubenswrapper[2556]: E0728 01:59:21.280950    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62\": container with ID starting with 008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62 not found: ID does not exist" containerID="008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62"
Jul 28 01:59:21.281027 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:21.280976    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62"} err="failed to get container status \"008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62\": rpc error: code = NotFound desc = could not find container \"008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62\": container with ID starting with 008cf7757194ca8ad7602061d4cb7cef7de0b1fe19dddd284783a8394abd6f62 not found: ID does not exist"
Jul 28 01:59:21.296453 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:21.296430    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"]
Jul 28 01:59:21.300569 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:21.300539    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewdsw9"]
Jul 28 01:59:22.286006 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:22.285974    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1db73cc-972b-418d-855c-b9d031b5af06" path="/var/lib/kubelet/pods/b1db73cc-972b-418d-855c-b9d031b5af06/volumes"
Jul 28 01:59:25.470626 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:25.470588    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:25.648131 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:25.648085    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:32.227993 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:32.227958    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:38.059196 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:38.059155    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:38.133114 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:38.133076    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:41.749127 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.749053    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:41.961705 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.961675    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"]
Jul 28 01:59:41.962090 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.962071    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf76b67-7df3-4c19-8a99-e6396eac1cf0" containerName="main"
Jul 28 01:59:41.962090 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.962090    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf76b67-7df3-4c19-8a99-e6396eac1cf0" containerName="main"
Jul 28 01:59:41.962294 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.962108    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1db73cc-972b-418d-855c-b9d031b5af06" containerName="main"
Jul 28 01:59:41.962294 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.962117    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1db73cc-972b-418d-855c-b9d031b5af06" containerName="main"
Jul 28 01:59:41.962294 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.962197    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf76b67-7df3-4c19-8a99-e6396eac1cf0" containerName="main"
Jul 28 01:59:41.962294 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.962210    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1db73cc-972b-418d-855c-b9d031b5af06" containerName="main"
Jul 28 01:59:41.970834 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.970811    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 01:59:41.974238 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.974209    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-qzvmm\""
Jul 28 01:59:41.974502 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.974484    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-pgwkj\""
Jul 28 01:59:41.974806 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.974521    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 01:59:41.974966 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.974547    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 01:59:41.975035 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.974620    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 28 01:59:41.975681 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:41.975660    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"]
Jul 28 01:59:42.015217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:42.015132    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj9rv\" (UniqueName: \"kubernetes.io/projected/938052d6-12db-4472-9a29-2c3e3adb04de-kube-api-access-jj9rv\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s\" (UID: \"938052d6-12db-4472-9a29-2c3e3adb04de\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 01:59:42.015217 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:42.015177    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/938052d6-12db-4472-9a29-2c3e3adb04de-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s\" (UID: \"938052d6-12db-4472-9a29-2c3e3adb04de\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 01:59:42.016367 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:42.016336    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:42.115807 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:42.115773    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj9rv\" (UniqueName: \"kubernetes.io/projected/938052d6-12db-4472-9a29-2c3e3adb04de-kube-api-access-jj9rv\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s\" (UID: \"938052d6-12db-4472-9a29-2c3e3adb04de\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 01:59:42.115967 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:42.115815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/938052d6-12db-4472-9a29-2c3e3adb04de-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s\" (UID: \"938052d6-12db-4472-9a29-2c3e3adb04de\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 01:59:42.118442 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:42.118416    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/938052d6-12db-4472-9a29-2c3e3adb04de-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s\" (UID: \"938052d6-12db-4472-9a29-2c3e3adb04de\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 01:59:42.124820 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:42.124787    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj9rv\" (UniqueName: \"kubernetes.io/projected/938052d6-12db-4472-9a29-2c3e3adb04de-kube-api-access-jj9rv\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s\" (UID: \"938052d6-12db-4472-9a29-2c3e3adb04de\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 01:59:42.283300 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:42.283223    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 01:59:42.418762 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:42.418739    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"]
Jul 28 01:59:42.420518 ip-10-0-140-37 kubenswrapper[2556]: W0728 01:59:42.420486    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938052d6_12db_4472_9a29_2c3e3adb04de.slice/crio-b75962ebccd5e2588204005c2493062de3f54f3a0c337bd61e1dc22a185ca81b WatchSource:0}: Error finding container b75962ebccd5e2588204005c2493062de3f54f3a0c337bd61e1dc22a185ca81b: Status 404 returned error can't find the container with id b75962ebccd5e2588204005c2493062de3f54f3a0c337bd61e1dc22a185ca81b
Jul 28 01:59:43.369816 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:43.369782    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s" event={"ID":"938052d6-12db-4472-9a29-2c3e3adb04de","Type":"ContainerStarted","Data":"31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee"}
Jul 28 01:59:43.370216 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:43.369833    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s" event={"ID":"938052d6-12db-4472-9a29-2c3e3adb04de","Type":"ContainerStarted","Data":"b75962ebccd5e2588204005c2493062de3f54f3a0c337bd61e1dc22a185ca81b"}
Jul 28 01:59:43.370216 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:43.369914    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 01:59:43.388545 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:43.388496    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s" podStartSLOduration=2.388483204 podStartE2EDuration="2.388483204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:59:43.387180602 +0000 UTC m=+2063.559925852" watchObservedRunningTime="2026-07-28 01:59:43.388483204 +0000 UTC m=+2063.561228453"
Jul 28 01:59:57.623572 ip-10-0-140-37 kubenswrapper[2556]: I0728 01:59:57.623522    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:00:14.375682 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:14.375644    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 02:00:19.270021 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.269987    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:00:19.460234 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.460201    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"]
Jul 28 02:00:19.464374 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.464354    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:19.468084 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.468060    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-zhrl2\""
Jul 28 02:00:19.468197 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.468092    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 28 02:00:19.475461 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.475435    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"]
Jul 28 02:00:19.493133 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.493110    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:00:19.544192 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.544134    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwff\" (UniqueName: \"kubernetes.io/projected/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-kube-api-access-xpwff\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw\" (UID: \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:19.544300 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.544187    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw\" (UID: \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:19.645487 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.645454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwff\" (UniqueName: \"kubernetes.io/projected/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-kube-api-access-xpwff\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw\" (UID: \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:19.645653 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.645524    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw\" (UID: \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:19.648021 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.648002    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw\" (UID: \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:19.653632 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.653614    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwff\" (UniqueName: \"kubernetes.io/projected/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-kube-api-access-xpwff\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw\" (UID: \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:19.776758 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.776726    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:19.909867 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:19.909844    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"]
Jul 28 02:00:19.912755 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:00:19.912715    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d7c2a9_2d9c_427b_b060_a1e11a2789b5.slice/crio-684d250b2bc3b2bc1be15fda6f3fef009594e0a7981620a7bdb5e7a6978b912d WatchSource:0}: Error finding container 684d250b2bc3b2bc1be15fda6f3fef009594e0a7981620a7bdb5e7a6978b912d: Status 404 returned error can't find the container with id 684d250b2bc3b2bc1be15fda6f3fef009594e0a7981620a7bdb5e7a6978b912d
Jul 28 02:00:20.392602 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:20.392578    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:00:20.395267 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:20.395248    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:00:20.541075 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:20.541046    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw" event={"ID":"14d7c2a9-2d9c-427b-b060-a1e11a2789b5","Type":"ContainerStarted","Data":"4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f"}
Jul 28 02:00:20.541234 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:20.541096    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw" event={"ID":"14d7c2a9-2d9c-427b-b060-a1e11a2789b5","Type":"ContainerStarted","Data":"684d250b2bc3b2bc1be15fda6f3fef009594e0a7981620a7bdb5e7a6978b912d"}
Jul 28 02:00:20.541234 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:20.541163    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:20.562981 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:20.562935    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw" podStartSLOduration=1.562921037 podStartE2EDuration="1.562921037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:00:20.561468042 +0000 UTC m=+2100.734213295" watchObservedRunningTime="2026-07-28 02:00:20.562921037 +0000 UTC m=+2100.735666285"
Jul 28 02:00:40.270024 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:40.269991    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:00:51.547475 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:51.547447    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:00:54.321031 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:54.321001    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:00:58.178142 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:58.178104    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"]
Jul 28 02:00:58.193730 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:58.193691    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:00:58.291939 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:00:58.291905    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:01.547641 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:01.547588    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw" podUID="14d7c2a9-2d9c-427b-b060-a1e11a2789b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:01:11.547502 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:11.547420    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw" podUID="14d7c2a9-2d9c-427b-b060-a1e11a2789b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:01:12.844519 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:12.844477    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:13.200251 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.200151    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw" podUID="14d7c2a9-2d9c-427b-b060-a1e11a2789b5" containerName="main" containerID="cri-o://4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f" gracePeriod=45
Jul 28 02:01:13.452983 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.452927    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:01:13.499876 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.499847    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpwff\" (UniqueName: \"kubernetes.io/projected/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-kube-api-access-xpwff\") pod \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\" (UID: \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\") "
Jul 28 02:01:13.499998 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.499883    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-tls-certs\") pod \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\" (UID: \"14d7c2a9-2d9c-427b-b060-a1e11a2789b5\") "
Jul 28 02:01:13.502243 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.502217    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14d7c2a9-2d9c-427b-b060-a1e11a2789b5" (UID: "14d7c2a9-2d9c-427b-b060-a1e11a2789b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:01:13.502342 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.502253    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-kube-api-access-xpwff" (OuterVolumeSpecName: "kube-api-access-xpwff") pod "14d7c2a9-2d9c-427b-b060-a1e11a2789b5" (UID: "14d7c2a9-2d9c-427b-b060-a1e11a2789b5"). InnerVolumeSpecName "kube-api-access-xpwff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:01:13.601064 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.601033    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwff\" (UniqueName: \"kubernetes.io/projected/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-kube-api-access-xpwff\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:01:13.601064 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.601061    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d7c2a9-2d9c-427b-b060-a1e11a2789b5-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:01:13.776356 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.776268    2556 generic.go:356] "Generic (PLEG): container finished" podID="14d7c2a9-2d9c-427b-b060-a1e11a2789b5" containerID="4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f" exitCode=0
Jul 28 02:01:13.776356 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.776318    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"
Jul 28 02:01:13.776356 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.776342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw" event={"ID":"14d7c2a9-2d9c-427b-b060-a1e11a2789b5","Type":"ContainerDied","Data":"4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f"}
Jul 28 02:01:13.776644 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.776419    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw" event={"ID":"14d7c2a9-2d9c-427b-b060-a1e11a2789b5","Type":"ContainerDied","Data":"684d250b2bc3b2bc1be15fda6f3fef009594e0a7981620a7bdb5e7a6978b912d"}
Jul 28 02:01:13.776644 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.776441    2556 scope.go:117] "RemoveContainer" containerID="4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f"
Jul 28 02:01:13.785373 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.785359    2556 scope.go:117] "RemoveContainer" containerID="4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f"
Jul 28 02:01:13.785633 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:01:13.785612    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f\": container with ID starting with 4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f not found: ID does not exist" containerID="4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f"
Jul 28 02:01:13.785699 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.785640    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f"} err="failed to get container status \"4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f\": rpc error: code = NotFound desc = could not find container \"4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f\": container with ID starting with 4d3e107fd6ef7ed1d670d41f9cfacbbb445335b1e9828660bca2a82d8ad82e3f not found: ID does not exist"
Jul 28 02:01:13.798958 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.798935    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"]
Jul 28 02:01:13.803615 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:13.803596    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8v2jw"]
Jul 28 02:01:14.287144 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:14.287111    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d7c2a9-2d9c-427b-b060-a1e11a2789b5" path="/var/lib/kubelet/pods/14d7c2a9-2d9c-427b-b060-a1e11a2789b5/volumes"
Jul 28 02:01:18.311414 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:18.311382    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:18.319978 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:18.319954    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"]
Jul 28 02:01:18.375990 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:18.375955    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:24.376115 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:24.376076    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s" podUID="938052d6-12db-4472-9a29-2c3e3adb04de" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:01:33.339200 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.339160    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s" podUID="938052d6-12db-4472-9a29-2c3e3adb04de" containerName="main" containerID="cri-o://31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee" gracePeriod=45
Jul 28 02:01:33.596603 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.596537    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 02:01:33.667729 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.667698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/938052d6-12db-4472-9a29-2c3e3adb04de-tls-certs\") pod \"938052d6-12db-4472-9a29-2c3e3adb04de\" (UID: \"938052d6-12db-4472-9a29-2c3e3adb04de\") "
Jul 28 02:01:33.667907 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.667742    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj9rv\" (UniqueName: \"kubernetes.io/projected/938052d6-12db-4472-9a29-2c3e3adb04de-kube-api-access-jj9rv\") pod \"938052d6-12db-4472-9a29-2c3e3adb04de\" (UID: \"938052d6-12db-4472-9a29-2c3e3adb04de\") "
Jul 28 02:01:33.669982 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.669941    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938052d6-12db-4472-9a29-2c3e3adb04de-kube-api-access-jj9rv" (OuterVolumeSpecName: "kube-api-access-jj9rv") pod "938052d6-12db-4472-9a29-2c3e3adb04de" (UID: "938052d6-12db-4472-9a29-2c3e3adb04de"). InnerVolumeSpecName "kube-api-access-jj9rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:01:33.669982 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.669941    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938052d6-12db-4472-9a29-2c3e3adb04de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "938052d6-12db-4472-9a29-2c3e3adb04de" (UID: "938052d6-12db-4472-9a29-2c3e3adb04de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:01:33.768746 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.768714    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj9rv\" (UniqueName: \"kubernetes.io/projected/938052d6-12db-4472-9a29-2c3e3adb04de-kube-api-access-jj9rv\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:01:33.768746 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.768738    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/938052d6-12db-4472-9a29-2c3e3adb04de-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:01:33.868392 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.868307    2556 generic.go:356] "Generic (PLEG): container finished" podID="938052d6-12db-4472-9a29-2c3e3adb04de" containerID="31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee" exitCode=0
Jul 28 02:01:33.868392 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.868374    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"
Jul 28 02:01:33.868596 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.868391    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s" event={"ID":"938052d6-12db-4472-9a29-2c3e3adb04de","Type":"ContainerDied","Data":"31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee"}
Jul 28 02:01:33.868596 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.868470    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s" event={"ID":"938052d6-12db-4472-9a29-2c3e3adb04de","Type":"ContainerDied","Data":"b75962ebccd5e2588204005c2493062de3f54f3a0c337bd61e1dc22a185ca81b"}
Jul 28 02:01:33.868596 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.868493    2556 scope.go:117] "RemoveContainer" containerID="31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee"
Jul 28 02:01:33.877004 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.876986    2556 scope.go:117] "RemoveContainer" containerID="31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee"
Jul 28 02:01:33.877257 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:01:33.877238    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee\": container with ID starting with 31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee not found: ID does not exist" containerID="31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee"
Jul 28 02:01:33.877331 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.877269    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee"} err="failed to get container status \"31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee\": rpc error: code = NotFound desc = could not find container \"31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee\": container with ID starting with 31fb3bf752d3ef4b421a4f04cf23e7e810a85546780783152dfeddf5b93434ee not found: ID does not exist"
Jul 28 02:01:33.893591 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.893543    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"]
Jul 28 02:01:33.897441 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:33.897419    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9x9s"]
Jul 28 02:01:34.286228 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:34.286136    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938052d6-12db-4472-9a29-2c3e3adb04de" path="/var/lib/kubelet/pods/938052d6-12db-4472-9a29-2c3e3adb04de/volumes"
Jul 28 02:01:38.357141 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:38.357103    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:39.903472 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:39.903429    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:44.539828 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:44.539791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:50.410221 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:50.410184    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:01:50.508963 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:01:50.508929    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:02:03.659025 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.658990    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:02:03.893023 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.892993    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"]
Jul 28 02:02:03.893340 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.893318    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d7c2a9-2d9c-427b-b060-a1e11a2789b5" containerName="main"
Jul 28 02:02:03.893340 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.893333    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d7c2a9-2d9c-427b-b060-a1e11a2789b5" containerName="main"
Jul 28 02:02:03.893496 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.893352    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938052d6-12db-4472-9a29-2c3e3adb04de" containerName="main"
Jul 28 02:02:03.893496 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.893357    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="938052d6-12db-4472-9a29-2c3e3adb04de" containerName="main"
Jul 28 02:02:03.893496 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.893422    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d7c2a9-2d9c-427b-b060-a1e11a2789b5" containerName="main"
Jul 28 02:02:03.893496 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.893432    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="938052d6-12db-4472-9a29-2c3e3adb04de" containerName="main"
Jul 28 02:02:03.896602 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.896581    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:02:03.899486 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.899467    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 28 02:02:03.900769 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.900744    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 28 02:02:03.900887 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.900744    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 28 02:02:03.900887 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.900850    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-4q6jr\""
Jul 28 02:02:03.901058 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.901041    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-drplq\""
Jul 28 02:02:03.909864 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.909816    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"]
Jul 28 02:02:03.909941 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.909926    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9965016-fadb-47a5-8e1f-7be202d4a8e6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w\" (UID: \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:02:03.909995 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.909984    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlnc\" (UniqueName: \"kubernetes.io/projected/c9965016-fadb-47a5-8e1f-7be202d4a8e6-kube-api-access-qwlnc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w\" (UID: \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:02:03.929264 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:03.929240    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:02:04.010629 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:04.010595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9965016-fadb-47a5-8e1f-7be202d4a8e6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w\" (UID: \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:02:04.010765 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:04.010671    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlnc\" (UniqueName: \"kubernetes.io/projected/c9965016-fadb-47a5-8e1f-7be202d4a8e6-kube-api-access-qwlnc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w\" (UID: \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:02:04.013330 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:04.013300    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9965016-fadb-47a5-8e1f-7be202d4a8e6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w\" (UID: \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:02:04.019782 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:04.019759    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlnc\" (UniqueName: \"kubernetes.io/projected/c9965016-fadb-47a5-8e1f-7be202d4a8e6-kube-api-access-qwlnc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w\" (UID: \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:02:04.207469 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:04.207390    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:02:04.336729 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:04.336706    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"]
Jul 28 02:02:04.338894 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:02:04.338868    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9965016_fadb_47a5_8e1f_7be202d4a8e6.slice/crio-6606bdb8ee560c01fe824eb71760d60df92b54352bee30bbb36d8f137538156c WatchSource:0}: Error finding container 6606bdb8ee560c01fe824eb71760d60df92b54352bee30bbb36d8f137538156c: Status 404 returned error can't find the container with id 6606bdb8ee560c01fe824eb71760d60df92b54352bee30bbb36d8f137538156c
Jul 28 02:02:05.001317 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:05.001280    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w" event={"ID":"c9965016-fadb-47a5-8e1f-7be202d4a8e6","Type":"ContainerStarted","Data":"e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0"}
Jul 28 02:02:05.001756 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:05.001342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w" event={"ID":"c9965016-fadb-47a5-8e1f-7be202d4a8e6","Type":"ContainerStarted","Data":"6606bdb8ee560c01fe824eb71760d60df92b54352bee30bbb36d8f137538156c"}
Jul 28 02:02:05.001756 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:05.001391    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:02:05.022255 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:05.022206    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w" podStartSLOduration=2.022189339 podStartE2EDuration="2.022189339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:02:05.019504196 +0000 UTC m=+2205.192249460" watchObservedRunningTime="2026-07-28 02:02:05.022189339 +0000 UTC m=+2205.194934597"
Jul 28 02:02:13.927007 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:13.926973    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:02:36.008132 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:02:36.008099    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:03:57.782049 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:03:57.782003    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:03:58.966519 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:03:58.966484    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"]
Jul 28 02:03:58.978081 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:03:58.978030    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:03:59.078832 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:03:59.078791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:04:06.007512 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:06.007470    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w" podUID="c9965016-fadb-47a5-8e1f-7be202d4a8e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:04:10.939280 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:10.939204    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:04:13.987828 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:13.987743    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w" podUID="c9965016-fadb-47a5-8e1f-7be202d4a8e6" containerName="main" containerID="cri-o://e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0" gracePeriod=45
Jul 28 02:04:14.247410 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.247351    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:04:14.318614 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.318583    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9965016-fadb-47a5-8e1f-7be202d4a8e6-tls-certs\") pod \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\" (UID: \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\") "
Jul 28 02:04:14.318801 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.318677    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwlnc\" (UniqueName: \"kubernetes.io/projected/c9965016-fadb-47a5-8e1f-7be202d4a8e6-kube-api-access-qwlnc\") pod \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\" (UID: \"c9965016-fadb-47a5-8e1f-7be202d4a8e6\") "
Jul 28 02:04:14.320872 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.320843    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9965016-fadb-47a5-8e1f-7be202d4a8e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9965016-fadb-47a5-8e1f-7be202d4a8e6" (UID: "c9965016-fadb-47a5-8e1f-7be202d4a8e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:04:14.321003 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.320939    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9965016-fadb-47a5-8e1f-7be202d4a8e6-kube-api-access-qwlnc" (OuterVolumeSpecName: "kube-api-access-qwlnc") pod "c9965016-fadb-47a5-8e1f-7be202d4a8e6" (UID: "c9965016-fadb-47a5-8e1f-7be202d4a8e6"). InnerVolumeSpecName "kube-api-access-qwlnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:04:14.420045 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.420015    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9965016-fadb-47a5-8e1f-7be202d4a8e6-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:04:14.420045 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.420040    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwlnc\" (UniqueName: \"kubernetes.io/projected/c9965016-fadb-47a5-8e1f-7be202d4a8e6-kube-api-access-qwlnc\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:04:14.584569 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.584533    2556 generic.go:356] "Generic (PLEG): container finished" podID="c9965016-fadb-47a5-8e1f-7be202d4a8e6" containerID="e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0" exitCode=0
Jul 28 02:04:14.584749 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.584594    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"
Jul 28 02:04:14.584749 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.584621    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w" event={"ID":"c9965016-fadb-47a5-8e1f-7be202d4a8e6","Type":"ContainerDied","Data":"e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0"}
Jul 28 02:04:14.584749 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.584684    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w" event={"ID":"c9965016-fadb-47a5-8e1f-7be202d4a8e6","Type":"ContainerDied","Data":"6606bdb8ee560c01fe824eb71760d60df92b54352bee30bbb36d8f137538156c"}
Jul 28 02:04:14.584749 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.584713    2556 scope.go:117] "RemoveContainer" containerID="e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0"
Jul 28 02:04:14.593714 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.593697    2556 scope.go:117] "RemoveContainer" containerID="e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0"
Jul 28 02:04:14.593957 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:04:14.593936    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0\": container with ID starting with e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0 not found: ID does not exist" containerID="e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0"
Jul 28 02:04:14.594023 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.593969    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0"} err="failed to get container status \"e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0\": rpc error: code = NotFound desc = could not find container \"e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0\": container with ID starting with e790f111aefba7fe5722329060f474633e295b78de3ec854caaa5d7dc3d9b1b0 not found: ID does not exist"
Jul 28 02:04:14.606950 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.606925    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"]
Jul 28 02:04:14.613623 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:14.613596    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schesmg8w"]
Jul 28 02:04:16.287018 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:16.286989    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9965016-fadb-47a5-8e1f-7be202d4a8e6" path="/var/lib/kubelet/pods/c9965016-fadb-47a5-8e1f-7be202d4a8e6/volumes"
Jul 28 02:04:18.758100 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:18.758063    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:04:18.852453 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:18.852414    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:04:24.778694 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:24.778658    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:04:30.465705 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:30.465662    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:04:30.583290 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:30.583258    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:04:38.751165 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.751133    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:04:38.984383 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.984345    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"]
Jul 28 02:04:38.984758 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.984744    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9965016-fadb-47a5-8e1f-7be202d4a8e6" containerName="main"
Jul 28 02:04:38.984809 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.984760    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9965016-fadb-47a5-8e1f-7be202d4a8e6" containerName="main"
Jul 28 02:04:38.984845 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.984821    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9965016-fadb-47a5-8e1f-7be202d4a8e6" containerName="main"
Jul 28 02:04:38.987892 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.987871    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:04:38.991694 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.991669    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-d5fgt\""
Jul 28 02:04:38.991694 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.991673    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 28 02:04:38.991951 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.991704    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-hf85n\""
Jul 28 02:04:38.991951 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.991711    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 28 02:04:38.991951 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.991700    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 28 02:04:38.996630 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:38.996608    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"]
Jul 28 02:04:39.021734 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.021659    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:04:39.131308 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.131272    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm2hz\" (UniqueName: \"kubernetes.io/projected/09acdfbe-39b3-4520-bfcf-1587011c497c-kube-api-access-nm2hz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8\" (UID: \"09acdfbe-39b3-4520-bfcf-1587011c497c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:04:39.131471 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.131316    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09acdfbe-39b3-4520-bfcf-1587011c497c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8\" (UID: \"09acdfbe-39b3-4520-bfcf-1587011c497c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:04:39.232372 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.232340    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm2hz\" (UniqueName: \"kubernetes.io/projected/09acdfbe-39b3-4520-bfcf-1587011c497c-kube-api-access-nm2hz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8\" (UID: \"09acdfbe-39b3-4520-bfcf-1587011c497c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:04:39.232541 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.232378    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09acdfbe-39b3-4520-bfcf-1587011c497c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8\" (UID: \"09acdfbe-39b3-4520-bfcf-1587011c497c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:04:39.235071 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.235041    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09acdfbe-39b3-4520-bfcf-1587011c497c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8\" (UID: \"09acdfbe-39b3-4520-bfcf-1587011c497c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:04:39.240710 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.240689    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm2hz\" (UniqueName: \"kubernetes.io/projected/09acdfbe-39b3-4520-bfcf-1587011c497c-kube-api-access-nm2hz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8\" (UID: \"09acdfbe-39b3-4520-bfcf-1587011c497c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:04:39.298057 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.297987    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:04:39.441176 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.441144    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"]
Jul 28 02:04:39.443957 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:04:39.443930    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09acdfbe_39b3_4520_bfcf_1587011c497c.slice/crio-68861f83319d250ff8248aed3eccf66c310c60e181e926ac9ca0eb7be6e3ace4 WatchSource:0}: Error finding container 68861f83319d250ff8248aed3eccf66c310c60e181e926ac9ca0eb7be6e3ace4: Status 404 returned error can't find the container with id 68861f83319d250ff8248aed3eccf66c310c60e181e926ac9ca0eb7be6e3ace4
Jul 28 02:04:39.695407 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.695374    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8" event={"ID":"09acdfbe-39b3-4520-bfcf-1587011c497c","Type":"ContainerStarted","Data":"23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468"}
Jul 28 02:04:39.695620 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.695426    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8" event={"ID":"09acdfbe-39b3-4520-bfcf-1587011c497c","Type":"ContainerStarted","Data":"68861f83319d250ff8248aed3eccf66c310c60e181e926ac9ca0eb7be6e3ace4"}
Jul 28 02:04:39.695620 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.695495    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:04:39.716056 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:39.716011    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8" podStartSLOduration=1.7159962279999998 podStartE2EDuration="1.715996228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:04:39.715707472 +0000 UTC m=+2359.888452722" watchObservedRunningTime="2026-07-28 02:04:39.715996228 +0000 UTC m=+2359.888741477"
Jul 28 02:04:50.426421 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:04:50.426381    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:05.508541 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.508505    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"]
Jul 28 02:05:05.518924 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:05:05.518889    2556 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 02:05:05.518924 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:05:05.518924 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 02:05:05.518924 ip-10-0-140-37 kubenswrapper[2556]:  > containerID="f00b2837d4fb9c8a601ecffcc6264eaef3f8e8e95630d94300c02184ec61a534" cmd=["/bin/sleep","15"]
Jul 28 02:05:05.519219 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.518942    2556 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv" message="" err=<
Jul 28 02:05:05.519219 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:05:05.519219 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 02:05:05.519219 ip-10-0-140-37 kubenswrapper[2556]:  >
Jul 28 02:05:05.519219 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:05:05.518954    2556 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 02:05:05.519219 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:05:05.519219 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 02:05:05.519219 ip-10-0-140-37 kubenswrapper[2556]:  > pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv" podUID="8e7a3387-a97f-4610-8872-e3ed7f6f885a" containerName="main" containerID="cri-o://f00b2837d4fb9c8a601ecffcc6264eaef3f8e8e95630d94300c02184ec61a534"
Jul 28 02:05:05.519219 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.519001    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv" podUID="8e7a3387-a97f-4610-8872-e3ed7f6f885a" containerName="main" containerID="cri-o://f00b2837d4fb9c8a601ecffcc6264eaef3f8e8e95630d94300c02184ec61a534" gracePeriod=60
Jul 28 02:05:05.525547 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.525516    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:05.631112 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.631055    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:05.813901 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.813877    2556 generic.go:356] "Generic (PLEG): container finished" podID="8e7a3387-a97f-4610-8872-e3ed7f6f885a" containerID="f00b2837d4fb9c8a601ecffcc6264eaef3f8e8e95630d94300c02184ec61a534" exitCode=0
Jul 28 02:05:05.814044 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.813938    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv" event={"ID":"8e7a3387-a97f-4610-8872-e3ed7f6f885a","Type":"ContainerDied","Data":"f00b2837d4fb9c8a601ecffcc6264eaef3f8e8e95630d94300c02184ec61a534"}
Jul 28 02:05:05.860965 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.860940    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 02:05:05.963466 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963436    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-dshm\") pod \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") "
Jul 28 02:05:05.963657 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963482    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rgqz\" (UniqueName: \"kubernetes.io/projected/8e7a3387-a97f-4610-8872-e3ed7f6f885a-kube-api-access-9rgqz\") pod \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") "
Jul 28 02:05:05.963657 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963527    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tmp-dir\") pod \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") "
Jul 28 02:05:05.963657 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963594    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-home\") pod \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") "
Jul 28 02:05:05.963657 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963655    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-model-cache\") pod \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") "
Jul 28 02:05:05.963829 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963707    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tls-certs\") pod \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\" (UID: \"8e7a3387-a97f-4610-8872-e3ed7f6f885a\") "
Jul 28 02:05:05.963892 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963810    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e7a3387-a97f-4610-8872-e3ed7f6f885a" (UID: "8e7a3387-a97f-4610-8872-e3ed7f6f885a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:05:05.963951 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963907    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-home" (OuterVolumeSpecName: "home") pod "8e7a3387-a97f-4610-8872-e3ed7f6f885a" (UID: "8e7a3387-a97f-4610-8872-e3ed7f6f885a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:05:05.963951 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963936    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e7a3387-a97f-4610-8872-e3ed7f6f885a" (UID: "8e7a3387-a97f-4610-8872-e3ed7f6f885a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:05:05.964066 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.963991    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tmp-dir\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:05:05.964066 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.964003    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-home\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:05:05.964066 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.964016    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-model-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:05:05.966334 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.966288    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e7a3387-a97f-4610-8872-e3ed7f6f885a" (UID: "8e7a3387-a97f-4610-8872-e3ed7f6f885a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:05:05.966453 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.966359    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-dshm" (OuterVolumeSpecName: "dshm") pod "8e7a3387-a97f-4610-8872-e3ed7f6f885a" (UID: "8e7a3387-a97f-4610-8872-e3ed7f6f885a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:05:05.966453 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:05.966389    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7a3387-a97f-4610-8872-e3ed7f6f885a-kube-api-access-9rgqz" (OuterVolumeSpecName: "kube-api-access-9rgqz") pod "8e7a3387-a97f-4610-8872-e3ed7f6f885a" (UID: "8e7a3387-a97f-4610-8872-e3ed7f6f885a"). InnerVolumeSpecName "kube-api-access-9rgqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:05:06.064944 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:06.064873    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7a3387-a97f-4610-8872-e3ed7f6f885a-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:05:06.064944 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:06.064898    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7a3387-a97f-4610-8872-e3ed7f6f885a-dshm\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:05:06.064944 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:06.064907    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rgqz\" (UniqueName: \"kubernetes.io/projected/8e7a3387-a97f-4610-8872-e3ed7f6f885a-kube-api-access-9rgqz\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:05:06.820619 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:06.820588    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"
Jul 28 02:05:06.821051 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:06.820593    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv" event={"ID":"8e7a3387-a97f-4610-8872-e3ed7f6f885a","Type":"ContainerDied","Data":"25b073951878782c9e0c286f83fff4df9cdb213b6cdc1fd9c9bb2ba5d15880b6"}
Jul 28 02:05:06.821051 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:06.820711    2556 scope.go:117] "RemoveContainer" containerID="f00b2837d4fb9c8a601ecffcc6264eaef3f8e8e95630d94300c02184ec61a534"
Jul 28 02:05:06.836889 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:06.836865    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"]
Jul 28 02:05:06.841200 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:06.841166    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv"]
Jul 28 02:05:08.286310 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:08.286273    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7a3387-a97f-4610-8872-e3ed7f6f885a" path="/var/lib/kubelet/pods/8e7a3387-a97f-4610-8872-e3ed7f6f885a/volumes"
Jul 28 02:05:10.702341 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:10.702313    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:05:20.418958 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:20.418926    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:05:20.423265 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:20.423246    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:05:25.706301 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:25.706269    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:25.804915 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:25.804885    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:36.956436 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:36.956393    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:37.381373 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.381345    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"]
Jul 28 02:05:37.381825 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.381803    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7a3387-a97f-4610-8872-e3ed7f6f885a" containerName="main"
Jul 28 02:05:37.381825 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.381822    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7a3387-a97f-4610-8872-e3ed7f6f885a" containerName="main"
Jul 28 02:05:37.382005 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.381893    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7a3387-a97f-4610-8872-e3ed7f6f885a" containerName="main"
Jul 28 02:05:37.384741 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.384721    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:05:37.388785 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.388759    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 28 02:05:37.388942 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.388835    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-epp-sa-dockercfg-6d85b\""
Jul 28 02:05:37.388942 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.388836    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 28 02:05:37.388942 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.388841    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 28 02:05:37.388942 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.388918    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-86qlh\""
Jul 28 02:05:37.397624 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.397601    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"]
Jul 28 02:05:37.430008 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.429970    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:37.435841 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.435818    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4\" (UID: \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:05:37.435942 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.435887    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwd2b\" (UniqueName: \"kubernetes.io/projected/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-kube-api-access-qwd2b\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4\" (UID: \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:05:37.536470 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.536441    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4\" (UID: \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:05:37.536664 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.536522    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwd2b\" (UniqueName: \"kubernetes.io/projected/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-kube-api-access-qwd2b\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4\" (UID: \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:05:37.539121 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.539100    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4\" (UID: \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:05:37.544918 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.544900    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwd2b\" (UniqueName: \"kubernetes.io/projected/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-kube-api-access-qwd2b\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4\" (UID: \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:05:37.697950 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.697866    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:05:37.833444 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.833358    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"]
Jul 28 02:05:37.835989 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:05:37.835965    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7c9c67_4034_4035_8d86_aeb0fbcd013a.slice/crio-e52eb15262360274c0a9bd014ad8ec42ed72dcafa9c2e7044909c1f22cfc9b93 WatchSource:0}: Error finding container e52eb15262360274c0a9bd014ad8ec42ed72dcafa9c2e7044909c1f22cfc9b93: Status 404 returned error can't find the container with id e52eb15262360274c0a9bd014ad8ec42ed72dcafa9c2e7044909c1f22cfc9b93
Jul 28 02:05:37.957676 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.957594    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4" event={"ID":"6c7c9c67-4034-4035-8d86-aeb0fbcd013a","Type":"ContainerStarted","Data":"754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8"}
Jul 28 02:05:37.957676 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.957660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4" event={"ID":"6c7c9c67-4034-4035-8d86-aeb0fbcd013a","Type":"ContainerStarted","Data":"e52eb15262360274c0a9bd014ad8ec42ed72dcafa9c2e7044909c1f22cfc9b93"}
Jul 28 02:05:37.958083 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.957714    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:05:37.981968 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:37.981921    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4" podStartSLOduration=0.981906248 podStartE2EDuration="981.906248ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:05:37.980224211 +0000 UTC m=+2418.152969462" watchObservedRunningTime="2026-07-28 02:05:37.981906248 +0000 UTC m=+2418.154651497"
Jul 28 02:05:48.592195 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:48.592158    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:48.694518 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:48.694463    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:50.248446 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:05:50.248406    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:08.964170 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:08.964138    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:06:11.244782 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:11.244747    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"]
Jul 28 02:06:11.245340 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:11.244856    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:11.340810 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:11.340771    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:13.906902 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:13.906858    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:13.963533 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:13.963499    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:20.702585 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:20.702531    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8" podUID="09acdfbe-39b3-4520-bfcf-1587011c497c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:06:26.264396 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:26.264357    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8" podUID="09acdfbe-39b3-4520-bfcf-1587011c497c" containerName="main" containerID="cri-o://23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468" gracePeriod=45
Jul 28 02:06:26.515447 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:26.515392    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:06:26.558380 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:26.558350    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm2hz\" (UniqueName: \"kubernetes.io/projected/09acdfbe-39b3-4520-bfcf-1587011c497c-kube-api-access-nm2hz\") pod \"09acdfbe-39b3-4520-bfcf-1587011c497c\" (UID: \"09acdfbe-39b3-4520-bfcf-1587011c497c\") "
Jul 28 02:06:26.558521 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:26.558391    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09acdfbe-39b3-4520-bfcf-1587011c497c-tls-certs\") pod \"09acdfbe-39b3-4520-bfcf-1587011c497c\" (UID: \"09acdfbe-39b3-4520-bfcf-1587011c497c\") "
Jul 28 02:06:26.560785 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:26.560758    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09acdfbe-39b3-4520-bfcf-1587011c497c-kube-api-access-nm2hz" (OuterVolumeSpecName: "kube-api-access-nm2hz") pod "09acdfbe-39b3-4520-bfcf-1587011c497c" (UID: "09acdfbe-39b3-4520-bfcf-1587011c497c"). InnerVolumeSpecName "kube-api-access-nm2hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:06:26.560884 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:26.560839    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09acdfbe-39b3-4520-bfcf-1587011c497c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09acdfbe-39b3-4520-bfcf-1587011c497c" (UID: "09acdfbe-39b3-4520-bfcf-1587011c497c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:06:26.659434 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:26.659399    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm2hz\" (UniqueName: \"kubernetes.io/projected/09acdfbe-39b3-4520-bfcf-1587011c497c-kube-api-access-nm2hz\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:06:26.659434 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:26.659430    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09acdfbe-39b3-4520-bfcf-1587011c497c-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:06:27.176250 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:27.176217    2556 generic.go:356] "Generic (PLEG): container finished" podID="09acdfbe-39b3-4520-bfcf-1587011c497c" containerID="23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468" exitCode=0
Jul 28 02:06:27.176436 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:27.176260    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"
Jul 28 02:06:27.176436 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:27.176293    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8" event={"ID":"09acdfbe-39b3-4520-bfcf-1587011c497c","Type":"ContainerDied","Data":"23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468"}
Jul 28 02:06:27.176436 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:27.176332    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8" event={"ID":"09acdfbe-39b3-4520-bfcf-1587011c497c","Type":"ContainerDied","Data":"68861f83319d250ff8248aed3eccf66c310c60e181e926ac9ca0eb7be6e3ace4"}
Jul 28 02:06:27.176436 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:27.176353    2556 scope.go:117] "RemoveContainer" containerID="23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468"
Jul 28 02:06:27.185258 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:27.185235    2556 scope.go:117] "RemoveContainer" containerID="23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468"
Jul 28 02:06:27.185511 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:06:27.185493    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468\": container with ID starting with 23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468 not found: ID does not exist" containerID="23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468"
Jul 28 02:06:27.185601 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:27.185522    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468"} err="failed to get container status \"23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468\": rpc error: code = NotFound desc = could not find container \"23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468\": container with ID starting with 23b045c068a978586b4d1bbadcd517a7b12d46bf8482403c7f36d4072a03d468 not found: ID does not exist"
Jul 28 02:06:27.197463 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:27.197438    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"]
Jul 28 02:06:27.201255 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:27.201232    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebm6k8"]
Jul 28 02:06:28.287355 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:28.287319    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09acdfbe-39b3-4520-bfcf-1587011c497c" path="/var/lib/kubelet/pods/09acdfbe-39b3-4520-bfcf-1587011c497c/volumes"
Jul 28 02:06:31.289697 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:31.289666    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:31.440856 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:31.440821    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:31.519044 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:31.519012    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:31.600822 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:31.600789    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:33.881378 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:33.881347    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:33.984175 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:33.984143    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:34.755575 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:34.755532    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:37.406999 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:37.406962    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:42.955876 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:42.955836    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:44.231178 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:44.231134    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:56.743342 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.743311    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:56.952311 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.952283    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"]
Jul 28 02:06:56.952710 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.952696    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09acdfbe-39b3-4520-bfcf-1587011c497c" containerName="main"
Jul 28 02:06:56.952767 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.952714    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="09acdfbe-39b3-4520-bfcf-1587011c497c" containerName="main"
Jul 28 02:06:56.952803 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.952774    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="09acdfbe-39b3-4520-bfcf-1587011c497c" containerName="main"
Jul 28 02:06:56.955834 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.955814    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:06:56.958344 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.958323    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 28 02:06:56.959617 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.959591    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-lqpkn\""
Jul 28 02:06:56.959722 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.959615    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 02:06:56.959722 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.959614    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-dpvh8\""
Jul 28 02:06:56.959860 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.959602    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 02:06:56.969209 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.969188    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"]
Jul 28 02:06:56.998073 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:56.997996    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:06:57.008972 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:57.008944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldm5r\" (UniqueName: \"kubernetes.io/projected/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-kube-api-access-ldm5r\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g\" (UID: \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:06:57.009082 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:57.008979    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g\" (UID: \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:06:57.110350 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:57.110319    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldm5r\" (UniqueName: \"kubernetes.io/projected/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-kube-api-access-ldm5r\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g\" (UID: \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:06:57.110350 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:57.110354    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g\" (UID: \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:06:57.113003 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:57.112981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g\" (UID: \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:06:57.119326 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:57.119302    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldm5r\" (UniqueName: \"kubernetes.io/projected/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-kube-api-access-ldm5r\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g\" (UID: \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:06:57.266238 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:57.266154    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:06:57.395509 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:57.395480    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"]
Jul 28 02:06:57.397897 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:06:57.397874    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370c8425_61f6_4da9_b3f0_bcc1c102d1d8.slice/crio-91757c46c55cb836b34c9d24023aa903ad836418ce13699a5474f517fca43b43 WatchSource:0}: Error finding container 91757c46c55cb836b34c9d24023aa903ad836418ce13699a5474f517fca43b43: Status 404 returned error can't find the container with id 91757c46c55cb836b34c9d24023aa903ad836418ce13699a5474f517fca43b43
Jul 28 02:06:58.317474 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:58.317437    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g" event={"ID":"370c8425-61f6-4da9-b3f0-bcc1c102d1d8","Type":"ContainerStarted","Data":"0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2"}
Jul 28 02:06:58.317946 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:58.317499    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:06:58.317946 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:58.317517    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g" event={"ID":"370c8425-61f6-4da9-b3f0-bcc1c102d1d8","Type":"ContainerStarted","Data":"91757c46c55cb836b34c9d24023aa903ad836418ce13699a5474f517fca43b43"}
Jul 28 02:06:58.334866 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:06:58.334809    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g" podStartSLOduration=2.334793549 podStartE2EDuration="2.334793549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:06:58.333215465 +0000 UTC m=+2498.505960711" watchObservedRunningTime="2026-07-28 02:06:58.334793549 +0000 UTC m=+2498.507538797"
Jul 28 02:07:01.344346 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:07:01.344311    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:07:29.323547 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:07:29.323519    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:07:37.437459 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:07:37.437416    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:04.609188 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:04.609153    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"]
Jul 28 02:09:04.643353 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:04.643323    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:04.745229 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:04.745180    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:09.323438 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:09.323398    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g" podUID="370c8425-61f6-4da9-b3f0-bcc1c102d1d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:09:19.323867 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.323829    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g" podUID="370c8425-61f6-4da9-b3f0-bcc1c102d1d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:09:19.628958 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.628861    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g" podUID="370c8425-61f6-4da9-b3f0-bcc1c102d1d8" containerName="main" containerID="cri-o://0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2" gracePeriod=45
Jul 28 02:09:19.886515 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.886460    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:09:19.936297 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.936269    2556 generic.go:356] "Generic (PLEG): container finished" podID="370c8425-61f6-4da9-b3f0-bcc1c102d1d8" containerID="0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2" exitCode=0
Jul 28 02:09:19.936431 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.936309    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"
Jul 28 02:09:19.936431 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.936357    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g" event={"ID":"370c8425-61f6-4da9-b3f0-bcc1c102d1d8","Type":"ContainerDied","Data":"0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2"}
Jul 28 02:09:19.936431 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.936403    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g" event={"ID":"370c8425-61f6-4da9-b3f0-bcc1c102d1d8","Type":"ContainerDied","Data":"91757c46c55cb836b34c9d24023aa903ad836418ce13699a5474f517fca43b43"}
Jul 28 02:09:19.936431 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.936427    2556 scope.go:117] "RemoveContainer" containerID="0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2"
Jul 28 02:09:19.944592 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.944545    2556 scope.go:117] "RemoveContainer" containerID="0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2"
Jul 28 02:09:19.944873 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:09:19.944854    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2\": container with ID starting with 0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2 not found: ID does not exist" containerID="0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2"
Jul 28 02:09:19.944968 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.944882    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2"} err="failed to get container status \"0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2\": rpc error: code = NotFound desc = could not find container \"0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2\": container with ID starting with 0e894a658e2fafa6333e41f917bf88a8c90bad52ea30f19a37b486ad74aa99a2 not found: ID does not exist"
Jul 28 02:09:19.994253 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.994235    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldm5r\" (UniqueName: \"kubernetes.io/projected/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-kube-api-access-ldm5r\") pod \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\" (UID: \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\") "
Jul 28 02:09:19.994362 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.994282    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-tls-certs\") pod \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\" (UID: \"370c8425-61f6-4da9-b3f0-bcc1c102d1d8\") "
Jul 28 02:09:19.996593 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.996540    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "370c8425-61f6-4da9-b3f0-bcc1c102d1d8" (UID: "370c8425-61f6-4da9-b3f0-bcc1c102d1d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:09:19.996688 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:19.996599    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-kube-api-access-ldm5r" (OuterVolumeSpecName: "kube-api-access-ldm5r") pod "370c8425-61f6-4da9-b3f0-bcc1c102d1d8" (UID: "370c8425-61f6-4da9-b3f0-bcc1c102d1d8"). InnerVolumeSpecName "kube-api-access-ldm5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:09:20.095553 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:20.095523    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldm5r\" (UniqueName: \"kubernetes.io/projected/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-kube-api-access-ldm5r\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:09:20.095553 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:20.095547    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/370c8425-61f6-4da9-b3f0-bcc1c102d1d8-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:09:20.259798 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:20.259770    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"]
Jul 28 02:09:20.264384 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:20.264362    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh5z4g"]
Jul 28 02:09:20.286800 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:20.286777    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="370c8425-61f6-4da9-b3f0-bcc1c102d1d8" path="/var/lib/kubelet/pods/370c8425-61f6-4da9-b3f0-bcc1c102d1d8/volumes"
Jul 28 02:09:24.569287 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:24.569250    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:24.665847 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:24.665806    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:24.773222 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:24.773191    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:24.956276 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:24.956246    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:27.101335 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:27.101297    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:27.202937 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:27.202892    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:27.930333 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:27.930276    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:30.783172 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:30.783139    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:30.850438 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:30.850374    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:30.918400 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:30.918367    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:31.052303 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:31.052228    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:31.153634 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:31.153602    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:36.497191 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:36.497154    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:36.596228 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:36.596196    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:55.703120 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.703083    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"]
Jul 28 02:09:55.703506 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.703437    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370c8425-61f6-4da9-b3f0-bcc1c102d1d8" containerName="main"
Jul 28 02:09:55.703506 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.703450    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="370c8425-61f6-4da9-b3f0-bcc1c102d1d8" containerName="main"
Jul 28 02:09:55.703616 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.703513    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="370c8425-61f6-4da9-b3f0-bcc1c102d1d8" containerName="main"
Jul 28 02:09:55.706312 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.706296    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.711436 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.711414    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 02:09:55.711595 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.711501    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-ws6bz\""
Jul 28 02:09:55.712178 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.712148    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 02:09:55.712289 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.712225    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 02:09:55.724273 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.724252    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"]
Jul 28 02:09:55.759864 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.759838    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:09:55.800763 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.800742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.800872 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.800810    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hpkv\" (UniqueName: \"kubernetes.io/projected/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-kube-api-access-4hpkv\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.800872 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.800855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.800970 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.800878    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.800970 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.800918    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.800970 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.800937    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.901871 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.901839    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpkv\" (UniqueName: \"kubernetes.io/projected/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-kube-api-access-4hpkv\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.901871 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.901874    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.902065 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.901891    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.902065 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.901914    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.902065 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.901934    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.902065 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.901964    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.902333 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.902310    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.902333 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.902322    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.902426 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.902393    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.904439 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.904417    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.904727 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.904711    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:55.913833 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:55.913810    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hpkv\" (UniqueName: \"kubernetes.io/projected/568c4ac2-0da4-4b97-8d6f-f9554a808bbb-kube-api-access-4hpkv\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" (UID: \"568c4ac2-0da4-4b97-8d6f-f9554a808bbb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:09:56.019258 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:09:56.019158    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:10:09.798272 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:10:09.798234    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:10:20.445506 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:10:20.445427    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:10:20.450091 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:10:20.450070    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:10:47.290515 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:10:47.290472    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:10:47.390668 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:10:47.390628    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:11:58.716969 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:11:58.716925    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:11:59.634752 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:11:59.634714    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:12:54.482378 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:12:54.482330    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:14:02.635959 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:14:02.635911    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:14:03.167227 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:14:03.167196    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:15:01.488237 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:15:01.488205    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:15:20.472705 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:15:20.472676    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:15:20.483097 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:15:20.483074    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:15:30.081195 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:15:30.081160    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:16:06.168608 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:16:06.168544    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:16:06.701844 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:16:06.701807    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:16:59.971844 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:16:59.971797    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:18:09.702646 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:18:09.702608    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:18:10.227311 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:18:10.227282    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:19:07.345202 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:19:07.345167    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:20:13.227881 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:20:13.227833    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:20:13.780533 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:13.780495    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:20:20.505181 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:20.505154    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:20:20.509799 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:20.509780    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:20:44.294262 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:44.294221    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"]
Jul 28 02:20:44.325543 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:44.325498    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:20:44.387350 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:44.387319    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:20:48.964134 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:48.964093    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4" podUID="6c7c9c67-4034-4035-8d86-aeb0fbcd013a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:20:58.963792 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:58.963756    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4" podUID="6c7c9c67-4034-4035-8d86-aeb0fbcd013a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:20:59.313553 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.313456    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4" podUID="6c7c9c67-4034-4035-8d86-aeb0fbcd013a" containerName="main" containerID="cri-o://754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8" gracePeriod=45
Jul 28 02:20:59.570053 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.569971    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:20:59.657670 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.657639    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-tls-certs\") pod \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\" (UID: \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\") "
Jul 28 02:20:59.657811 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.657684    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwd2b\" (UniqueName: \"kubernetes.io/projected/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-kube-api-access-qwd2b\") pod \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\" (UID: \"6c7c9c67-4034-4035-8d86-aeb0fbcd013a\") "
Jul 28 02:20:59.659988 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.659962    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c7c9c67-4034-4035-8d86-aeb0fbcd013a" (UID: "6c7c9c67-4034-4035-8d86-aeb0fbcd013a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:20:59.660161 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.660136    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-kube-api-access-qwd2b" (OuterVolumeSpecName: "kube-api-access-qwd2b") pod "6c7c9c67-4034-4035-8d86-aeb0fbcd013a" (UID: "6c7c9c67-4034-4035-8d86-aeb0fbcd013a"). InnerVolumeSpecName "kube-api-access-qwd2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:20:59.759120 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.759090    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:20:59.759120 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.759116    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwd2b\" (UniqueName: \"kubernetes.io/projected/6c7c9c67-4034-4035-8d86-aeb0fbcd013a-kube-api-access-qwd2b\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:20:59.986517 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.986478    2556 generic.go:356] "Generic (PLEG): container finished" podID="6c7c9c67-4034-4035-8d86-aeb0fbcd013a" containerID="754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8" exitCode=0
Jul 28 02:20:59.986517 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.986518    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"
Jul 28 02:20:59.987176 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.986532    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4" event={"ID":"6c7c9c67-4034-4035-8d86-aeb0fbcd013a","Type":"ContainerDied","Data":"754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8"}
Jul 28 02:20:59.987176 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.986648    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4" event={"ID":"6c7c9c67-4034-4035-8d86-aeb0fbcd013a","Type":"ContainerDied","Data":"e52eb15262360274c0a9bd014ad8ec42ed72dcafa9c2e7044909c1f22cfc9b93"}
Jul 28 02:20:59.987176 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.986685    2556 scope.go:117] "RemoveContainer" containerID="754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8"
Jul 28 02:20:59.996884 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.996707    2556 scope.go:117] "RemoveContainer" containerID="754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8"
Jul 28 02:20:59.997089 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:20:59.997064    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8\": container with ID starting with 754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8 not found: ID does not exist" containerID="754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8"
Jul 28 02:20:59.997197 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:20:59.997096    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8"} err="failed to get container status \"754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8\": rpc error: code = NotFound desc = could not find container \"754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8\": container with ID starting with 754c4fbc91f29cbf64830c3c3bdb84481f91b4fca9303225fc0ae28a2ae386b8 not found: ID does not exist"
Jul 28 02:21:00.010404 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:00.010366    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"]
Jul 28 02:21:00.013244 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:00.013220    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4"]
Jul 28 02:21:00.287305 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:00.287224    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c7c9c67-4034-4035-8d86-aeb0fbcd013a" path="/var/lib/kubelet/pods/6c7c9c67-4034-4035-8d86-aeb0fbcd013a/volumes"
Jul 28 02:21:02.479244 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:02.479208    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:21:02.579337 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:02.579302    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:21:26.063752 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.063715    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:21:26.162184 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.162147    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:21:26.218276 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.218242    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 02:21:26.218614 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.218600    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7c9c67-4034-4035-8d86-aeb0fbcd013a" containerName="main"
Jul 28 02:21:26.218672 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.218615    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7c9c67-4034-4035-8d86-aeb0fbcd013a" containerName="main"
Jul 28 02:21:26.218712 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.218673    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7c9c67-4034-4035-8d86-aeb0fbcd013a" containerName="main"
Jul 28 02:21:26.228197 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.228163    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.232165 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.232131    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 02:21:26.232330 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.232176    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 28 02:21:26.232330 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.232176    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 28 02:21:26.232330 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.232275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-5bgns\""
Jul 28 02:21:26.232330 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.232279    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 28 02:21:26.232585 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.232267    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-bg5dm\""
Jul 28 02:21:26.252272 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.252245    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:21:26.311709 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.311675    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"]
Jul 28 02:21:26.315138 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.315085    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:21:26.317677 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.317657    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-epp-sa-dockercfg-hpt8j\""
Jul 28 02:21:26.324532 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.324508    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"]
Jul 28 02:21:26.331367 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.331339    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:21:26.380971 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.380929    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7hx8\" (UniqueName: \"kubernetes.io/projected/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-kube-api-access-w7hx8\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.381147 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.381046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.381147 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.381118    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.381147 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.381145    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.381258 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.381164    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.381258 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.381189    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.481948 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.481891    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.481948 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.481945    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.482229 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.481977    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77vr\" (UniqueName: \"kubernetes.io/projected/cab3acc7-f70a-4be4-a99f-96751074c37e-kube-api-access-v77vr\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td\" (UID: \"cab3acc7-f70a-4be4-a99f-96751074c37e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:21:26.482229 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.482036    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7hx8\" (UniqueName: \"kubernetes.io/projected/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-kube-api-access-w7hx8\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.482229 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.482067    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.482229 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.482084    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3acc7-f70a-4be4-a99f-96751074c37e-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td\" (UID: \"cab3acc7-f70a-4be4-a99f-96751074c37e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:21:26.482229 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.482109    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.482229 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.482131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.482518 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.482333    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.482518 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.482374    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.482518 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.482422    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.484427 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.484398    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.484625 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.484609    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.490812 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.490791    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7hx8\" (UniqueName: \"kubernetes.io/projected/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-kube-api-access-w7hx8\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.540925 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.540886    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:21:26.583307 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.583272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77vr\" (UniqueName: \"kubernetes.io/projected/cab3acc7-f70a-4be4-a99f-96751074c37e-kube-api-access-v77vr\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td\" (UID: \"cab3acc7-f70a-4be4-a99f-96751074c37e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:21:26.583487 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.583372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3acc7-f70a-4be4-a99f-96751074c37e-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td\" (UID: \"cab3acc7-f70a-4be4-a99f-96751074c37e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:21:26.585887 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.585866    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3acc7-f70a-4be4-a99f-96751074c37e-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td\" (UID: \"cab3acc7-f70a-4be4-a99f-96751074c37e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:21:26.594736 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.594714    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77vr\" (UniqueName: \"kubernetes.io/projected/cab3acc7-f70a-4be4-a99f-96751074c37e-kube-api-access-v77vr\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td\" (UID: \"cab3acc7-f70a-4be4-a99f-96751074c37e\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:21:26.625956 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.625924    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:21:26.681749 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.681423    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 02:21:26.684194 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:21:26.684167    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb5cc5f_1206_4b07_a97f_2aaa80746fb8.slice/crio-4b48ce8737b8ba3d926a8c1ec62273df1b0fdfe06e09eb2dc5c4ce9ad0c96a3b WatchSource:0}: Error finding container 4b48ce8737b8ba3d926a8c1ec62273df1b0fdfe06e09eb2dc5c4ce9ad0c96a3b: Status 404 returned error can't find the container with id 4b48ce8737b8ba3d926a8c1ec62273df1b0fdfe06e09eb2dc5c4ce9ad0c96a3b
Jul 28 02:21:26.767050 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:26.767024    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"]
Jul 28 02:21:26.769858 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:21:26.769838    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab3acc7_f70a_4be4_a99f_96751074c37e.slice/crio-3e58344ce945b8f043a012e7cfb1b6ec39bc3044be9157863f30154443332918 WatchSource:0}: Error finding container 3e58344ce945b8f043a012e7cfb1b6ec39bc3044be9157863f30154443332918: Status 404 returned error can't find the container with id 3e58344ce945b8f043a012e7cfb1b6ec39bc3044be9157863f30154443332918
Jul 28 02:21:27.106317 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:27.106280    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td" event={"ID":"cab3acc7-f70a-4be4-a99f-96751074c37e","Type":"ContainerStarted","Data":"c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d"}
Jul 28 02:21:27.106818 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:27.106424    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:21:27.106818 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:27.106460    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td" event={"ID":"cab3acc7-f70a-4be4-a99f-96751074c37e","Type":"ContainerStarted","Data":"3e58344ce945b8f043a012e7cfb1b6ec39bc3044be9157863f30154443332918"}
Jul 28 02:21:27.107924 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:27.107902    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8","Type":"ContainerStarted","Data":"7dc83b74658d4b2cd45e17594e9de3fc0d5bb215b6c24c2627351add03be1952"}
Jul 28 02:21:27.108074 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:27.107946    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8","Type":"ContainerStarted","Data":"4b48ce8737b8ba3d926a8c1ec62273df1b0fdfe06e09eb2dc5c4ce9ad0c96a3b"}
Jul 28 02:21:27.124284 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:27.124227    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td" podStartSLOduration=1.124209451 podStartE2EDuration="1.124209451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:21:27.12393417 +0000 UTC m=+3367.296679421" watchObservedRunningTime="2026-07-28 02:21:27.124209451 +0000 UTC m=+3367.296954702"
Jul 28 02:21:27.147398 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:27.147338    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podStartSLOduration=1.147320266 podStartE2EDuration="1.147320266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:21:27.146180029 +0000 UTC m=+3367.318925277" watchObservedRunningTime="2026-07-28 02:21:27.147320266 +0000 UTC m=+3367.320065517"
Jul 28 02:21:47.100797 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:47.100761    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:21:58.115147 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:21:58.115115    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:22:16.781398 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:22:16.781295    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:22:17.334261 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:22:17.334228    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:23:09.246269 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:23:09.246235    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:23:44.305258 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:23:44.305224    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:23:44.406045 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:23:44.405998    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:24:20.335519 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:24:20.335467    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:24:20.886439 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:24:20.886408    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:24:47.387339 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:24:47.387303    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:24:48.902832 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:24:48.902795    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:24:49.825693 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:24:49.825656    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:24:49.927151 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:24:49.927111    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:24:50.615638 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:24:50.615602    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:24:54.932944 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:24:54.932899    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:25:05.397203 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:25:05.397165    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:25:20.531003 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:25:20.530968    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:25:20.537935 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:25:20.537913    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:25:24.427720 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:25:24.427669    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:25:24.524184 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:25:24.524147    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:25:24.595498 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:25:24.595466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:25:24.773604 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:25:24.773496    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:25:48.026174 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:25:48.026137    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:26:23.887047 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:26:23.887001    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:26:24.428397 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:26:24.428365    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:28:27.429090 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:28:27.429010    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:28:27.983746 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:28:27.983715    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:30:20.557788 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:30:20.557755    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:30:20.565341 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:30:20.565320    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:30:30.985246 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:30:30.985193    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:30:31.538667 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:30:31.538637    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:32:34.539733 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:32:34.539688    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:32:35.089656 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:32:35.089621    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:33:23.388183 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:33:23.388146    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:33:24.707294 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:33:24.707243    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:34:24.958091 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:34:24.958000    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:34:38.090594 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:34:38.090533    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:34:38.630316 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:34:38.630283    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:35:20.583756 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:35:20.583718    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:35:20.591847 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:35:20.591826    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:36:22.253503 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:22.253463    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"]
Jul 28 02:36:22.271386 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:22.271354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:36:22.325181 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:22.325142    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:36:22.931243 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:22.931201    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 02:36:22.945772 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:36:22.945724    2556 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 02:36:22.945772 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:36:22.945772 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 02:36:22.945772 ip-10-0-140-37 kubenswrapper[2556]:  > containerID="7dc83b74658d4b2cd45e17594e9de3fc0d5bb215b6c24c2627351add03be1952" cmd=["/bin/sleep","15"]
Jul 28 02:36:22.946079 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:22.945788    2556 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" message="" err=<
Jul 28 02:36:22.946079 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:36:22.946079 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 02:36:22.946079 ip-10-0-140-37 kubenswrapper[2556]:  >
Jul 28 02:36:22.946079 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:36:22.945801    2556 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 02:36:22.946079 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:36:22.946079 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 02:36:22.946079 ip-10-0-140-37 kubenswrapper[2556]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" containerName="main" containerID="cri-o://7dc83b74658d4b2cd45e17594e9de3fc0d5bb215b6c24c2627351add03be1952"
Jul 28 02:36:22.946079 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:22.945845    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" containerName="main" containerID="cri-o://7dc83b74658d4b2cd45e17594e9de3fc0d5bb215b6c24c2627351add03be1952" gracePeriod=60
Jul 28 02:36:23.117449 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.117415    2556 generic.go:356] "Generic (PLEG): container finished" podID="0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" containerID="7dc83b74658d4b2cd45e17594e9de3fc0d5bb215b6c24c2627351add03be1952" exitCode=0
Jul 28 02:36:23.117661 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.117471    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8","Type":"ContainerDied","Data":"7dc83b74658d4b2cd45e17594e9de3fc0d5bb215b6c24c2627351add03be1952"}
Jul 28 02:36:23.202728 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.202702    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:36:23.327053 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327025    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-dshm\") pod \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") "
Jul 28 02:36:23.327443 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327070    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7hx8\" (UniqueName: \"kubernetes.io/projected/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-kube-api-access-w7hx8\") pod \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") "
Jul 28 02:36:23.327443 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327095    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tmp-dir\") pod \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") "
Jul 28 02:36:23.327443 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327226    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-home\") pod \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") "
Jul 28 02:36:23.327443 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327281    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tls-certs\") pod \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") "
Jul 28 02:36:23.327443 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327334    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-model-cache\") pod \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\" (UID: \"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8\") "
Jul 28 02:36:23.327443 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327340    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" (UID: "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:36:23.327823 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327492    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-home" (OuterVolumeSpecName: "home") pod "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" (UID: "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:36:23.327823 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327637    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tmp-dir\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:36:23.327823 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327652    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-home\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:36:23.327823 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.327690    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" (UID: "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:36:23.329494 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.329472    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-kube-api-access-w7hx8" (OuterVolumeSpecName: "kube-api-access-w7hx8") pod "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" (UID: "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8"). InnerVolumeSpecName "kube-api-access-w7hx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:36:23.329601 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.329474    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" (UID: "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:36:23.329644 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.329592    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-dshm" (OuterVolumeSpecName: "dshm") pod "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" (UID: "0fb5cc5f-1206-4b07-a97f-2aaa80746fb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:36:23.428431 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.428408    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-model-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:36:23.428431 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.428429    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-dshm\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:36:23.428604 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.428439    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7hx8\" (UniqueName: \"kubernetes.io/projected/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-kube-api-access-w7hx8\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:36:23.428604 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:23.428448    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:36:24.123310 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:24.123284    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 02:36:24.123500 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:24.123326    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"0fb5cc5f-1206-4b07-a97f-2aaa80746fb8","Type":"ContainerDied","Data":"4b48ce8737b8ba3d926a8c1ec62273df1b0fdfe06e09eb2dc5c4ce9ad0c96a3b"}
Jul 28 02:36:24.123500 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:24.123382    2556 scope.go:117] "RemoveContainer" containerID="7dc83b74658d4b2cd45e17594e9de3fc0d5bb215b6c24c2627351add03be1952"
Jul 28 02:36:24.145507 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:24.145478    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 02:36:24.151029 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:24.151003    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 02:36:24.287028 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:24.286999    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" path="/var/lib/kubelet/pods/0fb5cc5f-1206-4b07-a97f-2aaa80746fb8/volumes"
Jul 28 02:36:28.114496 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:28.114459    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td" podUID="cab3acc7-f70a-4be4-a99f-96751074c37e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:36:37.273948 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:37.273884    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td" podUID="cab3acc7-f70a-4be4-a99f-96751074c37e" containerName="main" containerID="cri-o://c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d" gracePeriod=45
Jul 28 02:36:37.527099 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:37.527038    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:36:37.658078 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:37.658043    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3acc7-f70a-4be4-a99f-96751074c37e-tls-certs\") pod \"cab3acc7-f70a-4be4-a99f-96751074c37e\" (UID: \"cab3acc7-f70a-4be4-a99f-96751074c37e\") "
Jul 28 02:36:37.658224 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:37.658120    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v77vr\" (UniqueName: \"kubernetes.io/projected/cab3acc7-f70a-4be4-a99f-96751074c37e-kube-api-access-v77vr\") pod \"cab3acc7-f70a-4be4-a99f-96751074c37e\" (UID: \"cab3acc7-f70a-4be4-a99f-96751074c37e\") "
Jul 28 02:36:37.660424 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:37.660393    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab3acc7-f70a-4be4-a99f-96751074c37e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cab3acc7-f70a-4be4-a99f-96751074c37e" (UID: "cab3acc7-f70a-4be4-a99f-96751074c37e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:36:37.660424 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:37.660410    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab3acc7-f70a-4be4-a99f-96751074c37e-kube-api-access-v77vr" (OuterVolumeSpecName: "kube-api-access-v77vr") pod "cab3acc7-f70a-4be4-a99f-96751074c37e" (UID: "cab3acc7-f70a-4be4-a99f-96751074c37e"). InnerVolumeSpecName "kube-api-access-v77vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:36:37.759064 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:37.759032    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v77vr\" (UniqueName: \"kubernetes.io/projected/cab3acc7-f70a-4be4-a99f-96751074c37e-kube-api-access-v77vr\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:36:37.759064 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:37.759061    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3acc7-f70a-4be4-a99f-96751074c37e-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:36:38.185287 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.185249    2556 generic.go:356] "Generic (PLEG): container finished" podID="cab3acc7-f70a-4be4-a99f-96751074c37e" containerID="c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d" exitCode=0
Jul 28 02:36:38.185474 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.185292    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"
Jul 28 02:36:38.185474 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.185332    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td" event={"ID":"cab3acc7-f70a-4be4-a99f-96751074c37e","Type":"ContainerDied","Data":"c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d"}
Jul 28 02:36:38.185474 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.185407    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td" event={"ID":"cab3acc7-f70a-4be4-a99f-96751074c37e","Type":"ContainerDied","Data":"3e58344ce945b8f043a012e7cfb1b6ec39bc3044be9157863f30154443332918"}
Jul 28 02:36:38.185474 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.185433    2556 scope.go:117] "RemoveContainer" containerID="c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d"
Jul 28 02:36:38.194024 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.194008    2556 scope.go:117] "RemoveContainer" containerID="c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d"
Jul 28 02:36:38.194259 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:36:38.194241    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d\": container with ID starting with c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d not found: ID does not exist" containerID="c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d"
Jul 28 02:36:38.194300 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.194266    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d"} err="failed to get container status \"c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d\": rpc error: code = NotFound desc = could not find container \"c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d\": container with ID starting with c09a44acfb681510118fad12ff851a8fb7050966252a78a959997009b690ab0d not found: ID does not exist"
Jul 28 02:36:38.205436 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.205414    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"]
Jul 28 02:36:38.209583 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.209537    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td"]
Jul 28 02:36:38.286439 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.286413    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab3acc7-f70a-4be4-a99f-96751074c37e" path="/var/lib/kubelet/pods/cab3acc7-f70a-4be4-a99f-96751074c37e/volumes"
Jul 28 02:36:38.790474 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.790445    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:36:38.867111 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:38.867082    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:36:41.631430 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:36:41.631388    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:36:42.203745 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:36:42.203715    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:37:09.759296 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.759264    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:37:09.830744 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.830708    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 02:37:09.831060 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.831047    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" containerName="main"
Jul 28 02:37:09.831103 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.831062    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" containerName="main"
Jul 28 02:37:09.831103 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.831082    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab3acc7-f70a-4be4-a99f-96751074c37e" containerName="main"
Jul 28 02:37:09.831103 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.831088    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab3acc7-f70a-4be4-a99f-96751074c37e" containerName="main"
Jul 28 02:37:09.831196 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.831135    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab3acc7-f70a-4be4-a99f-96751074c37e" containerName="main"
Jul 28 02:37:09.831196 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.831144    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb5cc5f-1206-4b07-a97f-2aaa80746fb8" containerName="main"
Jul 28 02:37:09.834212 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.834190    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:09.837945 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.837887    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-t4mf7\""
Jul 28 02:37:09.838092 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.837966    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 28 02:37:09.838092 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.838004    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 28 02:37:09.838092 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.838042    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-dlfz9\""
Jul 28 02:37:09.838092 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.838066    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:37:09.838287 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.838167    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 28 02:37:09.847415 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.847393    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 02:37:09.933343 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.933315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:09.933534 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.933362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:09.933534 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.933429    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j24p\" (UniqueName: \"kubernetes.io/projected/948dca17-1012-4a7c-8f6c-d8f81c4073a8-kube-api-access-5j24p\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:09.933534 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.933465    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:09.933534 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.933527    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:09.933750 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:09.933596    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.034850 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.034765    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.034850 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.034800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.034850 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.034821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.035108 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.034876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.035108 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.034930    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j24p\" (UniqueName: \"kubernetes.io/projected/948dca17-1012-4a7c-8f6c-d8f81c4073a8-kube-api-access-5j24p\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.035108 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.034958    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.035257 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.035178    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.035257 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.035226    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.035345 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.035284    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.037317 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.037298    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.037437 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.037421    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.042437 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.042415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j24p\" (UniqueName: \"kubernetes.io/projected/948dca17-1012-4a7c-8f6c-d8f81c4073a8-kube-api-access-5j24p\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.077977 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.077950    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"]
Jul 28 02:37:10.081401 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.081381    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:10.083946 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.083927    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-epp-sa-dockercfg-rb74l\""
Jul 28 02:37:10.094729 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.094709    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"]
Jul 28 02:37:10.116581 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.116530    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:37:10.135718 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.135695    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fbcg\" (UniqueName: \"kubernetes.io/projected/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-kube-api-access-6fbcg\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd\" (UID: \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:10.135813 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.135738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd\" (UID: \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:10.144667 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.144648    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:10.236239 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.236190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fbcg\" (UniqueName: \"kubernetes.io/projected/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-kube-api-access-6fbcg\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd\" (UID: \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:10.236370 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.236275    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd\" (UID: \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:10.239138 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.239110    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd\" (UID: \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:10.244666 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.244634    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fbcg\" (UniqueName: \"kubernetes.io/projected/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-kube-api-access-6fbcg\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd\" (UID: \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:10.277741 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.277715    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 02:37:10.279918 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:37:10.279890    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948dca17_1012_4a7c_8f6c_d8f81c4073a8.slice/crio-6e5094eeb446bfe2528afc0b4903a6642c2708d579e3122ec20f22880e27a7a3 WatchSource:0}: Error finding container 6e5094eeb446bfe2528afc0b4903a6642c2708d579e3122ec20f22880e27a7a3: Status 404 returned error can't find the container with id 6e5094eeb446bfe2528afc0b4903a6642c2708d579e3122ec20f22880e27a7a3
Jul 28 02:37:10.336114 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.336087    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"948dca17-1012-4a7c-8f6c-d8f81c4073a8","Type":"ContainerStarted","Data":"6e5094eeb446bfe2528afc0b4903a6642c2708d579e3122ec20f22880e27a7a3"}
Jul 28 02:37:10.391689 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.391654    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:10.524049 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:10.524024    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"]
Jul 28 02:37:10.525911 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:37:10.525882    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8488c6bd_988a_4cb6_aa66_3e33bcc5ba83.slice/crio-74223c5c986c07de057a8c172fb8fa361796da28cbed350bba1366966411c35d WatchSource:0}: Error finding container 74223c5c986c07de057a8c172fb8fa361796da28cbed350bba1366966411c35d: Status 404 returned error can't find the container with id 74223c5c986c07de057a8c172fb8fa361796da28cbed350bba1366966411c35d
Jul 28 02:37:11.342663 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:11.342623    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"948dca17-1012-4a7c-8f6c-d8f81c4073a8","Type":"ContainerStarted","Data":"b5ba82d8d2fec54f162ca07018a116a9d1e13ea545f741448052382e179f969f"}
Jul 28 02:37:11.344165 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:11.344134    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd" event={"ID":"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83","Type":"ContainerStarted","Data":"71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170"}
Jul 28 02:37:11.344286 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:11.344173    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd" event={"ID":"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83","Type":"ContainerStarted","Data":"74223c5c986c07de057a8c172fb8fa361796da28cbed350bba1366966411c35d"}
Jul 28 02:37:11.344286 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:11.344203    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:11.359969 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:11.359928    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podStartSLOduration=2.3599155769999998 podStartE2EDuration="2.359915577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:37:11.358147255 +0000 UTC m=+4311.530892505" watchObservedRunningTime="2026-07-28 02:37:11.359915577 +0000 UTC m=+4311.532660826"
Jul 28 02:37:11.374599 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:11.374541    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd" podStartSLOduration=1.374526629 podStartE2EDuration="1.374526629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:37:11.373319847 +0000 UTC m=+4311.546065158" watchObservedRunningTime="2026-07-28 02:37:11.374526629 +0000 UTC m=+4311.547271878"
Jul 28 02:37:12.145099 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:12.145064    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:20.144908 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:20.144878    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:20.157757 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:20.157729    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:20.396367 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:20.396282    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:37:42.350460 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:42.350429    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:37:53.220211 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:37:53.220156    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:38:07.237966 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:38:07.237928    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:38:45.204670 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:38:45.204630    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:38:45.771735 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:38:45.771706    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:40:10.642877 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:10.642844    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:12.361177 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:12.361094    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:12.872240 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:12.872210    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:12.971045 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:12.971003    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:13.083907 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:13.083869    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:18.540128 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:18.540091    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:18.601809 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:18.601779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:20.140598 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:20.140552    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:20.241142 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:20.241115    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:20.341919 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:20.341888    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:20.610294 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:20.610269    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:40:20.624800 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:20.624779    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:40:24.389741 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:24.389708    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:24.490311 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:24.490283    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:24.589788 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:24.589757    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:26.242712 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:26.242673    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:26.339812 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:26.339781    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:29.330521 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:29.330487    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:29.578439 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:29.578409    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:29.678194 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:29.678099    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:30.634287 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:30.634252    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:30.734170 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:30.734134    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:34.332171 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:34.332140    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:34.409806 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:34.409771    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:34.515266 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:34.515232    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:35.413512 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:35.413482    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:35.514604 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:35.514552    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:35.832184 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:35.832145    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:35.931274 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:35.931237    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:36.033749 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:36.033721    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:36.335600 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:36.335545    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:36.451154 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:36.451126    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:36.569028 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:36.568988    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:37.833711 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:37.833679    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:37.934703 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:37.934672    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:38.036019 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:38.035979    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:39.028512 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:39.028467    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:39.129239 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:39.129214    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:39.228813 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:39.228783    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:40.206788 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:40.206750    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:40.323966 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:40.323935    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:40.421862 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:40.421828    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:40.521665 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:40.521585    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:41.638747 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:41.638720    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:41.739812 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:41.739782    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:42.760485 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:42.760453    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:42.864267 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:42.864239    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:43.971277 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:43.971245    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:44.029182 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:44.029152    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:44.735937 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:44.735904    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:44.801192 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:44.801160    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:46.207219 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:46.207185    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:46.307457 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:46.307430    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:40:48.772200 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:40:48.772155    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:40:49.315572 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:40:49.315532    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:41:01.957629 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:01.957593    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:41:02.118903 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.118871    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"]
Jul 28 02:41:02.122425 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.122404    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:41:02.125137 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.125109    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 02:41:02.125263 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.125156    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 02:41:02.126503 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.126483    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-qq5wh\""
Jul 28 02:41:02.126637 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.126484    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-r9p95\""
Jul 28 02:41:02.126637 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.126597    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 02:41:02.132521 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.132497    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"]
Jul 28 02:41:02.151455 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.151427    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:41:02.251080 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.250997    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf\" (UID: \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:41:02.251080 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.251043    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9z9g\" (UniqueName: \"kubernetes.io/projected/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-kube-api-access-r9z9g\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf\" (UID: \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:41:02.351681 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.351646    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf\" (UID: \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:41:02.351828 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.351695    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9z9g\" (UniqueName: \"kubernetes.io/projected/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-kube-api-access-r9z9g\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf\" (UID: \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:41:02.354315 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.354287    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf\" (UID: \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:41:02.359157 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.359132    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9z9g\" (UniqueName: \"kubernetes.io/projected/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-kube-api-access-r9z9g\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf\" (UID: \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:41:02.434765 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.434721    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:41:02.563099 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:02.563072    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"]
Jul 28 02:41:02.564917 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:41:02.564885    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72d2c0e_b323_4087_b9f1_6a3921fb9ada.slice/crio-5b2d571d5ce551552921d6229e9c36e5c393febd228e4051aa0ee95b21b4ca56 WatchSource:0}: Error finding container 5b2d571d5ce551552921d6229e9c36e5c393febd228e4051aa0ee95b21b4ca56: Status 404 returned error can't find the container with id 5b2d571d5ce551552921d6229e9c36e5c393febd228e4051aa0ee95b21b4ca56
Jul 28 02:41:03.378806 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:03.378764    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf" event={"ID":"e72d2c0e-b323-4087-b9f1-6a3921fb9ada","Type":"ContainerStarted","Data":"fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153"}
Jul 28 02:41:03.378806 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:03.378810    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf" event={"ID":"e72d2c0e-b323-4087-b9f1-6a3921fb9ada","Type":"ContainerStarted","Data":"5b2d571d5ce551552921d6229e9c36e5c393febd228e4051aa0ee95b21b4ca56"}
Jul 28 02:41:03.379265 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:03.378832    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:41:03.395136 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:03.395083    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf" podStartSLOduration=1.3950662409999999 podStartE2EDuration="1.395066241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:41:03.394171876 +0000 UTC m=+4543.566917126" watchObservedRunningTime="2026-07-28 02:41:03.395066241 +0000 UTC m=+4543.567811489"
Jul 28 02:41:29.050834 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:29.050792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:41:34.389006 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:41:34.388975    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:42:31.252063 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:42:31.252032    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:42:52.317054 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:42:52.317013    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:42:52.446945 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:42:52.446910    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:42:52.530735 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:42:52.530706    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:42:52.864105 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:42:52.864074    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:43:44.835060 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:43:44.835021    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"]
Jul 28 02:43:44.837456 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:43:44.837429    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:43:44.937444 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:43:44.937401    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:43:44.989935 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:43:44.989902    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:43:54.388890 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:43:54.388854    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf" podUID="e72d2c0e-b323-4087-b9f1-6a3921fb9ada" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:43:59.855431 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:43:59.855391    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf" podUID="e72d2c0e-b323-4087-b9f1-6a3921fb9ada" containerName="main" containerID="cri-o://fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153" gracePeriod=45
Jul 28 02:44:00.107908 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.107848    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:44:00.173336 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.173299    2556 generic.go:356] "Generic (PLEG): container finished" podID="e72d2c0e-b323-4087-b9f1-6a3921fb9ada" containerID="fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153" exitCode=0
Jul 28 02:44:00.173532 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.173344    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"
Jul 28 02:44:00.173532 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.173361    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf" event={"ID":"e72d2c0e-b323-4087-b9f1-6a3921fb9ada","Type":"ContainerDied","Data":"fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153"}
Jul 28 02:44:00.173532 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.173417    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf" event={"ID":"e72d2c0e-b323-4087-b9f1-6a3921fb9ada","Type":"ContainerDied","Data":"5b2d571d5ce551552921d6229e9c36e5c393febd228e4051aa0ee95b21b4ca56"}
Jul 28 02:44:00.173532 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.173445    2556 scope.go:117] "RemoveContainer" containerID="fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153"
Jul 28 02:44:00.181806 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.181785    2556 scope.go:117] "RemoveContainer" containerID="fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153"
Jul 28 02:44:00.182125 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:44:00.182098    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153\": container with ID starting with fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153 not found: ID does not exist" containerID="fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153"
Jul 28 02:44:00.182203 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.182133    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153"} err="failed to get container status \"fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153\": rpc error: code = NotFound desc = could not find container \"fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153\": container with ID starting with fd726275a2caad70f15dfc1f20223dd9ef1a90f860cc57942f34a8195fc74153 not found: ID does not exist"
Jul 28 02:44:00.203814 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.203787    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-tls-certs\") pod \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\" (UID: \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\") "
Jul 28 02:44:00.203949 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.203824    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9z9g\" (UniqueName: \"kubernetes.io/projected/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-kube-api-access-r9z9g\") pod \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\" (UID: \"e72d2c0e-b323-4087-b9f1-6a3921fb9ada\") "
Jul 28 02:44:00.206268 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.206233    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e72d2c0e-b323-4087-b9f1-6a3921fb9ada" (UID: "e72d2c0e-b323-4087-b9f1-6a3921fb9ada"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:44:00.206372 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.206305    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-kube-api-access-r9z9g" (OuterVolumeSpecName: "kube-api-access-r9z9g") pod "e72d2c0e-b323-4087-b9f1-6a3921fb9ada" (UID: "e72d2c0e-b323-4087-b9f1-6a3921fb9ada"). InnerVolumeSpecName "kube-api-access-r9z9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:44:00.305203 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.305167    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:44:00.305203 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.305197    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9z9g\" (UniqueName: \"kubernetes.io/projected/e72d2c0e-b323-4087-b9f1-6a3921fb9ada-kube-api-access-r9z9g\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:44:00.488898 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.488813    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"]
Jul 28 02:44:00.493080 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:00.493050    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-zf5pf"]
Jul 28 02:44:02.286762 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:02.286726    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e72d2c0e-b323-4087-b9f1-6a3921fb9ada" path="/var/lib/kubelet/pods/e72d2c0e-b323-4087-b9f1-6a3921fb9ada/volumes"
Jul 28 02:44:08.050545 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.050512    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:44:08.300168 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.300138    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"]
Jul 28 02:44:08.300485 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.300474    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e72d2c0e-b323-4087-b9f1-6a3921fb9ada" containerName="main"
Jul 28 02:44:08.300541 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.300487    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e72d2c0e-b323-4087-b9f1-6a3921fb9ada" containerName="main"
Jul 28 02:44:08.300634 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.300544    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e72d2c0e-b323-4087-b9f1-6a3921fb9ada" containerName="main"
Jul 28 02:44:08.304984 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.304963    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:08.311581 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.310740    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-pszr9\""
Jul 28 02:44:08.311581 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.311036    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 02:44:08.311581 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.311116    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 02:44:08.311581 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.311246    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 02:44:08.311581 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.311363    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-r9p95\""
Jul 28 02:44:08.322897 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.322876    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"]
Jul 28 02:44:08.369060 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.369037    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:44:08.385353 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.385331    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q\" (UID: \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:08.385505 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.385485    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6kgh\" (UniqueName: \"kubernetes.io/projected/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-kube-api-access-n6kgh\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q\" (UID: \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:08.486120 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.486089    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6kgh\" (UniqueName: \"kubernetes.io/projected/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-kube-api-access-n6kgh\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q\" (UID: \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:08.486251 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.486134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q\" (UID: \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:08.488914 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.488892    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q\" (UID: \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:08.499864 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.499838    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6kgh\" (UniqueName: \"kubernetes.io/projected/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-kube-api-access-n6kgh\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q\" (UID: \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:08.620284 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.620253    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:08.751407 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:08.751383    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"]
Jul 28 02:44:08.753803 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:44:08.753774    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff19af1b_ae74_424d_8f5a_ef780ab5d6d9.slice/crio-20b9f06a125c37621830b2e331bd5036cb18918fcf4d1c02f843439ebe846c2c WatchSource:0}: Error finding container 20b9f06a125c37621830b2e331bd5036cb18918fcf4d1c02f843439ebe846c2c: Status 404 returned error can't find the container with id 20b9f06a125c37621830b2e331bd5036cb18918fcf4d1c02f843439ebe846c2c
Jul 28 02:44:09.218874 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:09.218838    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q" event={"ID":"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9","Type":"ContainerStarted","Data":"2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c"}
Jul 28 02:44:09.219305 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:09.218889    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q" event={"ID":"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9","Type":"ContainerStarted","Data":"20b9f06a125c37621830b2e331bd5036cb18918fcf4d1c02f843439ebe846c2c"}
Jul 28 02:44:09.219305 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:09.218958    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:09.236480 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:09.236424    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q" podStartSLOduration=1.236404648 podStartE2EDuration="1.236404648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:44:09.235105604 +0000 UTC m=+4729.407850856" watchObservedRunningTime="2026-07-28 02:44:09.236404648 +0000 UTC m=+4729.409149898"
Jul 28 02:44:40.224940 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:40.224911    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:44:54.055663 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:54.055619    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:44:55.865454 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:44:55.865412    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:44:56.422964 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:44:56.422927    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:45:20.637173 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:45:20.637144    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:45:20.653084 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:45:20.653058    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:46:38.087526 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:38.087487    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"]
Jul 28 02:46:38.126524 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:38.126457    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:46:39.388602 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:39.388547    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:46:40.225249 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:40.225207    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q" podUID="ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:46:50.224681 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:50.224634    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q" podUID="ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:46:51.757461 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:51.757425    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:46:53.107600 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.107534    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q" podUID="ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" containerName="main" containerID="cri-o://2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c" gracePeriod=45
Jul 28 02:46:53.368487 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.368424    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:46:53.413354 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.413329    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-tls-certs\") pod \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\" (UID: \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\") "
Jul 28 02:46:53.413484 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.413364    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6kgh\" (UniqueName: \"kubernetes.io/projected/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-kube-api-access-n6kgh\") pod \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\" (UID: \"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9\") "
Jul 28 02:46:53.415506 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.415474    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" (UID: "ff19af1b-ae74-424d-8f5a-ef780ab5d6d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:46:53.415644 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.415578    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-kube-api-access-n6kgh" (OuterVolumeSpecName: "kube-api-access-n6kgh") pod "ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" (UID: "ff19af1b-ae74-424d-8f5a-ef780ab5d6d9"). InnerVolumeSpecName "kube-api-access-n6kgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:46:53.514330 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.514298    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:46:53.514330 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.514326    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6kgh\" (UniqueName: \"kubernetes.io/projected/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9-kube-api-access-n6kgh\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:46:53.944898 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.944857    2556 generic.go:356] "Generic (PLEG): container finished" podID="ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" containerID="2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c" exitCode=0
Jul 28 02:46:53.945101 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.944904    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"
Jul 28 02:46:53.945101 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.944942    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q" event={"ID":"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9","Type":"ContainerDied","Data":"2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c"}
Jul 28 02:46:53.945101 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.945005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q" event={"ID":"ff19af1b-ae74-424d-8f5a-ef780ab5d6d9","Type":"ContainerDied","Data":"20b9f06a125c37621830b2e331bd5036cb18918fcf4d1c02f843439ebe846c2c"}
Jul 28 02:46:53.945101 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.945029    2556 scope.go:117] "RemoveContainer" containerID="2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c"
Jul 28 02:46:53.954086 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.953885    2556 scope.go:117] "RemoveContainer" containerID="2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c"
Jul 28 02:46:53.954179 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:46:53.954156    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c\": container with ID starting with 2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c not found: ID does not exist" containerID="2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c"
Jul 28 02:46:53.954179 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.954188    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c"} err="failed to get container status \"2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c\": rpc error: code = NotFound desc = could not find container \"2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c\": container with ID starting with 2c8ebb86e805fb8099b36f906010fea62733f46be4e3261aef28a769cc5e543c not found: ID does not exist"
Jul 28 02:46:53.965031 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.965004    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"]
Jul 28 02:46:53.968692 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:53.968669    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-ggf5q"]
Jul 28 02:46:54.287130 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:54.287050    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" path="/var/lib/kubelet/pods/ff19af1b-ae74-424d-8f5a-ef780ab5d6d9/volumes"
Jul 28 02:46:58.529267 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:58.529229    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:46:58.629993 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:58.629952    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:46:58.729304 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:58.729262    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:46:59.424583 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:46:59.424519    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:46:59.973691 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:46:59.973658    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:47:04.473188 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:47:04.473141    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:47:10.249132 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:47:10.249096    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:47:10.347126 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:47:10.347098    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:47:21.074245 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:47:21.074211    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:48:39.784311 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:48:39.784274    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:49:02.974309 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:49:02.974266    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:49:03.533818 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:49:03.533784    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:49:54.995985 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:49:54.995945    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:50:12.946910 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:12.946862    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:50:13.051821 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:13.051784    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:50:13.152016 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:13.151977    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:50:19.041619 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:19.041581    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:50:19.105791 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:19.105764    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:50:20.663311 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:20.663282    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:50:20.679494 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:20.679467    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:50:27.851247 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:27.851214    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:50:41.854324 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:41.854286    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:50:41.954475 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:50:41.954437    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:51:06.534266 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:51:06.534220    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:51:07.074626 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:51:07.074588    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:52:06.097990 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.097955    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 02:52:06.103248 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.103221    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"]
Jul 28 02:52:06.107741 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:52:06.107712    2556 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 02:52:06.107741 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:52:06.107741 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 02:52:06.107741 ip-10-0-140-37 kubenswrapper[2556]:  > containerID="b5ba82d8d2fec54f162ca07018a116a9d1e13ea545f741448052382e179f969f" cmd=["/bin/sleep","15"]
Jul 28 02:52:06.107982 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.107757    2556 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" message="" err=<
Jul 28 02:52:06.107982 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:52:06.107982 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 02:52:06.107982 ip-10-0-140-37 kubenswrapper[2556]:  >
Jul 28 02:52:06.107982 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:52:06.107770    2556 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 02:52:06.107982 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:52:06.107982 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 02:52:06.107982 ip-10-0-140-37 kubenswrapper[2556]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="948dca17-1012-4a7c-8f6c-d8f81c4073a8" containerName="main" containerID="cri-o://b5ba82d8d2fec54f162ca07018a116a9d1e13ea545f741448052382e179f969f"
Jul 28 02:52:06.107982 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.107800    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="948dca17-1012-4a7c-8f6c-d8f81c4073a8" containerName="main" containerID="cri-o://b5ba82d8d2fec54f162ca07018a116a9d1e13ea545f741448052382e179f969f" gracePeriod=60
Jul 28 02:52:06.352131 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.352042    2556 generic.go:356] "Generic (PLEG): container finished" podID="948dca17-1012-4a7c-8f6c-d8f81c4073a8" containerID="b5ba82d8d2fec54f162ca07018a116a9d1e13ea545f741448052382e179f969f" exitCode=0
Jul 28 02:52:06.352131 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.352114    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"948dca17-1012-4a7c-8f6c-d8f81c4073a8","Type":"ContainerDied","Data":"b5ba82d8d2fec54f162ca07018a116a9d1e13ea545f741448052382e179f969f"}
Jul 28 02:52:06.449270 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.449249    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:52:06.563587 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.563541    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-dshm\") pod \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") "
Jul 28 02:52:06.563755 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.563637    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tmp-dir\") pod \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") "
Jul 28 02:52:06.563755 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.563660    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-home\") pod \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") "
Jul 28 02:52:06.563755 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.563683    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tls-certs\") pod \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") "
Jul 28 02:52:06.563755 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.563749    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-model-cache\") pod \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") "
Jul 28 02:52:06.563970 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.563796    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j24p\" (UniqueName: \"kubernetes.io/projected/948dca17-1012-4a7c-8f6c-d8f81c4073a8-kube-api-access-5j24p\") pod \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\" (UID: \"948dca17-1012-4a7c-8f6c-d8f81c4073a8\") "
Jul 28 02:52:06.563970 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.563917    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-home" (OuterVolumeSpecName: "home") pod "948dca17-1012-4a7c-8f6c-d8f81c4073a8" (UID: "948dca17-1012-4a7c-8f6c-d8f81c4073a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:06.563970 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.563934    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "948dca17-1012-4a7c-8f6c-d8f81c4073a8" (UID: "948dca17-1012-4a7c-8f6c-d8f81c4073a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:06.564128 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.564017    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "948dca17-1012-4a7c-8f6c-d8f81c4073a8" (UID: "948dca17-1012-4a7c-8f6c-d8f81c4073a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:06.564128 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.564107    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-model-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:52:06.564128 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.564123    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tmp-dir\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:52:06.564243 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.564133    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-home\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:52:06.566020 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.565995    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-dshm" (OuterVolumeSpecName: "dshm") pod "948dca17-1012-4a7c-8f6c-d8f81c4073a8" (UID: "948dca17-1012-4a7c-8f6c-d8f81c4073a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:06.566117 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.566085    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948dca17-1012-4a7c-8f6c-d8f81c4073a8-kube-api-access-5j24p" (OuterVolumeSpecName: "kube-api-access-5j24p") pod "948dca17-1012-4a7c-8f6c-d8f81c4073a8" (UID: "948dca17-1012-4a7c-8f6c-d8f81c4073a8"). InnerVolumeSpecName "kube-api-access-5j24p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:52:06.566117 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.566110    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "948dca17-1012-4a7c-8f6c-d8f81c4073a8" (UID: "948dca17-1012-4a7c-8f6c-d8f81c4073a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:52:06.664965 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.664883    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948dca17-1012-4a7c-8f6c-d8f81c4073a8-dshm\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:52:06.664965 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.664914    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948dca17-1012-4a7c-8f6c-d8f81c4073a8-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:52:06.664965 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.664924    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j24p\" (UniqueName: \"kubernetes.io/projected/948dca17-1012-4a7c-8f6c-d8f81c4073a8-kube-api-access-5j24p\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:52:06.680968 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:06.680937    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:52:07.364242 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:07.364208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"948dca17-1012-4a7c-8f6c-d8f81c4073a8","Type":"ContainerDied","Data":"6e5094eeb446bfe2528afc0b4903a6642c2708d579e3122ec20f22880e27a7a3"}
Jul 28 02:52:07.364242 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:07.364240    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 02:52:07.364740 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:07.364293    2556 scope.go:117] "RemoveContainer" containerID="b5ba82d8d2fec54f162ca07018a116a9d1e13ea545f741448052382e179f969f"
Jul 28 02:52:07.385836 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:07.385807    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 02:52:07.385836 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:07.385814    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="948dca17-1012-4a7c-8f6c-d8f81c4073a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Jul 28 02:52:07.390890 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:07.390869    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 02:52:07.905024 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:07.904988    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:52:08.286400 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:08.286371    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="948dca17-1012-4a7c-8f6c-d8f81c4073a8" path="/var/lib/kubelet/pods/948dca17-1012-4a7c-8f6c-d8f81c4073a8/volumes"
Jul 28 02:52:12.350011 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:12.349930    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd" podUID="8488c6bd-988a-4cb6-aa66-3e33bcc5ba83" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:52:13.362847 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:13.362811    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:52:19.291395 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:19.291351    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:52:19.391956 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:19.391928    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:52:21.122361 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.122321    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd" podUID="8488c6bd-988a-4cb6-aa66-3e33bcc5ba83" containerName="main" containerID="cri-o://71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170" gracePeriod=45
Jul 28 02:52:21.373071 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.373045    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:52:21.432444 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.432408    2556 generic.go:356] "Generic (PLEG): container finished" podID="8488c6bd-988a-4cb6-aa66-3e33bcc5ba83" containerID="71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170" exitCode=0
Jul 28 02:52:21.432629 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.432470    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"
Jul 28 02:52:21.432629 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.432496    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd" event={"ID":"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83","Type":"ContainerDied","Data":"71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170"}
Jul 28 02:52:21.432629 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.432593    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd" event={"ID":"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83","Type":"ContainerDied","Data":"74223c5c986c07de057a8c172fb8fa361796da28cbed350bba1366966411c35d"}
Jul 28 02:52:21.432629 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.432620    2556 scope.go:117] "RemoveContainer" containerID="71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170"
Jul 28 02:52:21.443126 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.443108    2556 scope.go:117] "RemoveContainer" containerID="71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170"
Jul 28 02:52:21.443582 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:52:21.443535    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170\": container with ID starting with 71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170 not found: ID does not exist" containerID="71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170"
Jul 28 02:52:21.443675 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.443584    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170"} err="failed to get container status \"71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170\": rpc error: code = NotFound desc = could not find container \"71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170\": container with ID starting with 71cfaa0dfe66453d140adbb4134cc3e154fa2fa44cb1a631e7d025bf7365d170 not found: ID does not exist"
Jul 28 02:52:21.491182 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.491106    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fbcg\" (UniqueName: \"kubernetes.io/projected/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-kube-api-access-6fbcg\") pod \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\" (UID: \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\") "
Jul 28 02:52:21.491323 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.491206    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-tls-certs\") pod \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\" (UID: \"8488c6bd-988a-4cb6-aa66-3e33bcc5ba83\") "
Jul 28 02:52:21.493247 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.493223    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8488c6bd-988a-4cb6-aa66-3e33bcc5ba83" (UID: "8488c6bd-988a-4cb6-aa66-3e33bcc5ba83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:52:21.493372 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.493352    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-kube-api-access-6fbcg" (OuterVolumeSpecName: "kube-api-access-6fbcg") pod "8488c6bd-988a-4cb6-aa66-3e33bcc5ba83" (UID: "8488c6bd-988a-4cb6-aa66-3e33bcc5ba83"). InnerVolumeSpecName "kube-api-access-6fbcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:52:21.592659 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.592621    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fbcg\" (UniqueName: \"kubernetes.io/projected/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-kube-api-access-6fbcg\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:52:21.592659 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.592654    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:52:21.753803 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.753728    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"]
Jul 28 02:52:21.756866 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:21.756841    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd"]
Jul 28 02:52:22.287135 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:22.287103    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8488c6bd-988a-4cb6-aa66-3e33bcc5ba83" path="/var/lib/kubelet/pods/8488c6bd-988a-4cb6-aa66-3e33bcc5ba83/volumes"
Jul 28 02:52:25.875182 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:25.875148    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:52:25.973680 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:52:25.973647    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:53:01.632697 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.632667    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"]
Jul 28 02:53:01.633193 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.633012    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="948dca17-1012-4a7c-8f6c-d8f81c4073a8" containerName="main"
Jul 28 02:53:01.633193 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.633026    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="948dca17-1012-4a7c-8f6c-d8f81c4073a8" containerName="main"
Jul 28 02:53:01.633193 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.633041    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" containerName="main"
Jul 28 02:53:01.633193 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.633047    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" containerName="main"
Jul 28 02:53:01.633193 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.633062    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8488c6bd-988a-4cb6-aa66-3e33bcc5ba83" containerName="main"
Jul 28 02:53:01.633193 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.633068    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8488c6bd-988a-4cb6-aa66-3e33bcc5ba83" containerName="main"
Jul 28 02:53:01.633193 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.633120    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="948dca17-1012-4a7c-8f6c-d8f81c4073a8" containerName="main"
Jul 28 02:53:01.633193 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.633127    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8488c6bd-988a-4cb6-aa66-3e33bcc5ba83" containerName="main"
Jul 28 02:53:01.633193 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.633134    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff19af1b-ae74-424d-8f5a-ef780ab5d6d9" containerName="main"
Jul 28 02:53:01.636196 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.636180    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.639822 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.639802    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"kube-root-ca.crt\""
Jul 28 02:53:01.639822 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.639811    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"default-dockercfg-bpqhr\""
Jul 28 02:53:01.640008 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.639811    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"openshift-service-ca.crt\""
Jul 28 02:53:01.640008 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.639976    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"autoscale-cleanup-hpa-kserve-self-signed-certs\""
Jul 28 02:53:01.647042 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.647022    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"]
Jul 28 02:53:01.660812 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.660785    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:53:01.715239 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.715210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febc0b4d-1902-4a04-853d-b525b0d87a47-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.715413 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.715244    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-dshm\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.715413 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.715270    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hwm4\" (UniqueName: \"kubernetes.io/projected/febc0b4d-1902-4a04-853d-b525b0d87a47-kube-api-access-6hwm4\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.715413 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.715323    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-model-cache\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.715413 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.715401    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-tmp-dir\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.715650 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.715470    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-home\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.816406 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.816369    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febc0b4d-1902-4a04-853d-b525b0d87a47-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.816406 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.816407    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-dshm\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.816698 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.816429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hwm4\" (UniqueName: \"kubernetes.io/projected/febc0b4d-1902-4a04-853d-b525b0d87a47-kube-api-access-6hwm4\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.816698 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.816460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-model-cache\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.816698 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.816492    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-tmp-dir\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.816698 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.816615    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-home\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.816948 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.816858    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-model-cache\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.816948 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.816923    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-tmp-dir\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.816948 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.816935    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-home\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.819051 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.819025    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-dshm\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.819271 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.819248    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febc0b4d-1902-4a04-853d-b525b0d87a47-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.826613 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.826593    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hwm4\" (UniqueName: \"kubernetes.io/projected/febc0b4d-1902-4a04-853d-b525b0d87a47-kube-api-access-6hwm4\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:01.849106 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.849079    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"]
Jul 28 02:53:01.852801 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.852784    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:01.855255 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.855239    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"autoscale-cleanup-hpa-epp-sa-dockercfg-f2ltd\""
Jul 28 02:53:01.861054 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.861034    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"]
Jul 28 02:53:01.874118 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.874091    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:53:01.917292 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.917215    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe510b7-a916-4b71-81dc-7e23ae6e3159-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq\" (UID: \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:01.917292 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.917266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkx4z\" (UniqueName: \"kubernetes.io/projected/2fe510b7-a916-4b71-81dc-7e23ae6e3159-kube-api-access-hkx4z\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq\" (UID: \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:01.948250 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:01.948222    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:02.017747 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.017717    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe510b7-a916-4b71-81dc-7e23ae6e3159-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq\" (UID: \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:02.017870 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.017776    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkx4z\" (UniqueName: \"kubernetes.io/projected/2fe510b7-a916-4b71-81dc-7e23ae6e3159-kube-api-access-hkx4z\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq\" (UID: \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:02.020527 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.020479    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe510b7-a916-4b71-81dc-7e23ae6e3159-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq\" (UID: \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:02.026624 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.026598    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkx4z\" (UniqueName: \"kubernetes.io/projected/2fe510b7-a916-4b71-81dc-7e23ae6e3159-kube-api-access-hkx4z\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq\" (UID: \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:02.079592 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.079545    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"]
Jul 28 02:53:02.081393 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:53:02.081363    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebc0b4d_1902_4a04_853d_b525b0d87a47.slice/crio-87e7c8697a3ca2ea9c1f727183a4661bd0150e1d0cd0ccad55005501a70507b5 WatchSource:0}: Error finding container 87e7c8697a3ca2ea9c1f727183a4661bd0150e1d0cd0ccad55005501a70507b5: Status 404 returned error can't find the container with id 87e7c8697a3ca2ea9c1f727183a4661bd0150e1d0cd0ccad55005501a70507b5
Jul 28 02:53:02.163516 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.163491    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:02.294171 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.294146    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"]
Jul 28 02:53:02.296538 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:53:02.296512    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe510b7_a916_4b71_81dc_7e23ae6e3159.slice/crio-f2dea12d4e7eeb67d234dd14401f9fad1acebb2727d8f3042369f960ccf49410 WatchSource:0}: Error finding container f2dea12d4e7eeb67d234dd14401f9fad1acebb2727d8f3042369f960ccf49410: Status 404 returned error can't find the container with id f2dea12d4e7eeb67d234dd14401f9fad1acebb2727d8f3042369f960ccf49410
Jul 28 02:53:02.614930 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.614890    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8" event={"ID":"febc0b4d-1902-4a04-853d-b525b0d87a47","Type":"ContainerStarted","Data":"75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4"}
Jul 28 02:53:02.615133 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.614956    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8" event={"ID":"febc0b4d-1902-4a04-853d-b525b0d87a47","Type":"ContainerStarted","Data":"87e7c8697a3ca2ea9c1f727183a4661bd0150e1d0cd0ccad55005501a70507b5"}
Jul 28 02:53:02.616366 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.616342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq" event={"ID":"2fe510b7-a916-4b71-81dc-7e23ae6e3159","Type":"ContainerStarted","Data":"3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37"}
Jul 28 02:53:02.616478 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.616385    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq" event={"ID":"2fe510b7-a916-4b71-81dc-7e23ae6e3159","Type":"ContainerStarted","Data":"f2dea12d4e7eeb67d234dd14401f9fad1acebb2727d8f3042369f960ccf49410"}
Jul 28 02:53:02.616478 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.616459    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:02.649248 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.649205    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq" podStartSLOduration=1.649193157 podStartE2EDuration="1.649193157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:53:02.647963647 +0000 UTC m=+5262.820708896" watchObservedRunningTime="2026-07-28 02:53:02.649193157 +0000 UTC m=+5262.821938405"
Jul 28 02:53:02.649916 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.649889    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8" podStartSLOduration=1.649882952 podStartE2EDuration="1.649882952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:53:02.632842962 +0000 UTC m=+5262.805588211" watchObservedRunningTime="2026-07-28 02:53:02.649882952 +0000 UTC m=+5262.822628200"
Jul 28 02:53:02.949044 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:02.948963    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:10.075864 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:53:10.075823    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:53:10.651377 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:10.651344    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:53:11.949319 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:11.949291    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:11.961875 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:11.961855    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:12.670997 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:12.670968    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 02:53:33.622777 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:33.622744    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 02:53:34.900090 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:34.900039    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:53:55.150620 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:55.150582    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:53:55.250585 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:55.250516    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:53:55.351251 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:53:55.351217    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:54:01.151932 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:01.151892    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:54:01.252897 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:01.252854    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:54:14.178894 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:14.178852    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:54:42.017466 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.017427    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:54:42.183488 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.183458    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"]
Jul 28 02:54:42.195819 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.195794    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:54:42.197178 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.197148    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"]
Jul 28 02:54:42.199997 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.199947    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 28 02:54:42.200131 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.200098    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-gkmdk\""
Jul 28 02:54:42.200201 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.200139    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-cs6rb\""
Jul 28 02:54:42.200266 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.200226    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 28 02:54:42.200266 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.200245    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 28 02:54:42.225742 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.225718    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:54:42.265105 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.265077    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqk2d\" (UniqueName: \"kubernetes.io/projected/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-kube-api-access-dqk2d\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-84mh8\" (UID: \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:54:42.265221 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.265114    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-84mh8\" (UID: \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:54:42.365963 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.365937    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqk2d\" (UniqueName: \"kubernetes.io/projected/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-kube-api-access-dqk2d\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-84mh8\" (UID: \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:54:42.366107 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.365974    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-84mh8\" (UID: \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:54:42.368514 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.368488    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-84mh8\" (UID: \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:54:42.376988 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.376962    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqk2d\" (UniqueName: \"kubernetes.io/projected/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-kube-api-access-dqk2d\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-84mh8\" (UID: \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:54:42.507627 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.507593    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:54:42.636044 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:42.636006    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"]
Jul 28 02:54:42.638196 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:54:42.638166    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe55e6c_2eb2_438f_9d67_9a2b9573ff29.slice/crio-49cdade16b7d2196437a98339464e7a52232e94acb65863dbea21c08fce72d97 WatchSource:0}: Error finding container 49cdade16b7d2196437a98339464e7a52232e94acb65863dbea21c08fce72d97: Status 404 returned error can't find the container with id 49cdade16b7d2196437a98339464e7a52232e94acb65863dbea21c08fce72d97
Jul 28 02:54:43.059157 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:43.059063    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8" event={"ID":"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29","Type":"ContainerStarted","Data":"8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce"}
Jul 28 02:54:43.059157 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:43.059118    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8" event={"ID":"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29","Type":"ContainerStarted","Data":"49cdade16b7d2196437a98339464e7a52232e94acb65863dbea21c08fce72d97"}
Jul 28 02:54:43.059678 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:43.059200    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:54:43.080456 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:54:43.080417    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8" podStartSLOduration=1.080403032 podStartE2EDuration="1.080403032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:54:43.079087206 +0000 UTC m=+5363.251832477" watchObservedRunningTime="2026-07-28 02:54:43.080403032 +0000 UTC m=+5363.253148280"
Jul 28 02:55:13.652581 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:55:13.652488    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:55:14.064886 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:55:14.064805    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:55:14.200301 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:55:14.200270    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:55:20.693958 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:55:20.693934    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:55:20.708108 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:55:20.708087    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 02:55:31.171817 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:55:31.171775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:55:37.431163 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:55:37.431125    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:56:15.191956 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:56:15.191921    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:07.340509 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:07.340471    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:07.341721 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:07.341696    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"]
Jul 28 02:57:07.404124 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:07.404076    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:14.065373 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:14.065331    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8" podUID="bfe55e6c-2eb2-438f-9d67-9a2b9573ff29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:57:17.201181 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:57:17.201137    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:57:17.745890 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:17.745862    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 02:57:20.945677 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:20.945635    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:22.361453 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.361403    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8" podUID="bfe55e6c-2eb2-438f-9d67-9a2b9573ff29" containerName="main" containerID="cri-o://8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce" gracePeriod=45
Jul 28 02:57:22.600545 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.600522    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:57:22.732729 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.732645    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-tls-certs\") pod \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\" (UID: \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\") "
Jul 28 02:57:22.732729 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.732711    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqk2d\" (UniqueName: \"kubernetes.io/projected/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-kube-api-access-dqk2d\") pod \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\" (UID: \"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29\") "
Jul 28 02:57:22.735000 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.734971    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bfe55e6c-2eb2-438f-9d67-9a2b9573ff29" (UID: "bfe55e6c-2eb2-438f-9d67-9a2b9573ff29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:57:22.735117 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.735069    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-kube-api-access-dqk2d" (OuterVolumeSpecName: "kube-api-access-dqk2d") pod "bfe55e6c-2eb2-438f-9d67-9a2b9573ff29" (UID: "bfe55e6c-2eb2-438f-9d67-9a2b9573ff29"). InnerVolumeSpecName "kube-api-access-dqk2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:57:22.768831 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.768806    2556 generic.go:356] "Generic (PLEG): container finished" podID="bfe55e6c-2eb2-438f-9d67-9a2b9573ff29" containerID="8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce" exitCode=0
Jul 28 02:57:22.768942 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.768848    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8" event={"ID":"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29","Type":"ContainerDied","Data":"8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce"}
Jul 28 02:57:22.768942 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.768885    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8" event={"ID":"bfe55e6c-2eb2-438f-9d67-9a2b9573ff29","Type":"ContainerDied","Data":"49cdade16b7d2196437a98339464e7a52232e94acb65863dbea21c08fce72d97"}
Jul 28 02:57:22.768942 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.768906    2556 scope.go:117] "RemoveContainer" containerID="8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce"
Jul 28 02:57:22.768942 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.768850    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"
Jul 28 02:57:22.778195 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.778176    2556 scope.go:117] "RemoveContainer" containerID="8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce"
Jul 28 02:57:22.778431 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:57:22.778413    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce\": container with ID starting with 8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce not found: ID does not exist" containerID="8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce"
Jul 28 02:57:22.778492 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.778441    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce"} err="failed to get container status \"8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce\": rpc error: code = NotFound desc = could not find container \"8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce\": container with ID starting with 8dd1efed82a1699efaaa8a21a6e8cf0badd8bfc94fc888f506be822891b0dcce not found: ID does not exist"
Jul 28 02:57:22.790539 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.790517    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"]
Jul 28 02:57:22.798196 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.798172    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-84mh8"]
Jul 28 02:57:22.833512 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.833487    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:57:22.833512 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:22.833507    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqk2d\" (UniqueName: \"kubernetes.io/projected/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29-kube-api-access-dqk2d\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 02:57:24.286165 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:24.286135    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe55e6c-2eb2-438f-9d67-9a2b9573ff29" path="/var/lib/kubelet/pods/bfe55e6c-2eb2-438f-9d67-9a2b9573ff29/volumes"
Jul 28 02:57:27.719290 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:27.719253    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:27.815877 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:27.815844    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:34.101547 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:34.101505    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:39.512637 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:39.512594    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:39.613469 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:39.613433    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:52.749732 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:52.749688    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:58.579070 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.579039    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:58.773708 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.773673    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"]
Jul 28 02:57:58.774034 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.774022    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe55e6c-2eb2-438f-9d67-9a2b9573ff29" containerName="main"
Jul 28 02:57:58.774078 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.774036    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe55e6c-2eb2-438f-9d67-9a2b9573ff29" containerName="main"
Jul 28 02:57:58.774113 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.774095    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe55e6c-2eb2-438f-9d67-9a2b9573ff29" containerName="main"
Jul 28 02:57:58.777093 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.777074    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:57:58.780830 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.780805    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 02:57:58.780830 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.780819    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 02:57:58.781037 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.780835    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-x5pll\""
Jul 28 02:57:58.781037 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.780816    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 02:57:58.781037 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.780823    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-psdvf\""
Jul 28 02:57:58.785130 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.785111    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"]
Jul 28 02:57:58.799803 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.799770    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:57:58.834800 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.834742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49acf685-8a66-4d5a-9277-f881ccc75343-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg\" (UID: \"49acf685-8a66-4d5a-9277-f881ccc75343\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:57:58.834909 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.834799    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w6p8\" (UniqueName: \"kubernetes.io/projected/49acf685-8a66-4d5a-9277-f881ccc75343-kube-api-access-6w6p8\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg\" (UID: \"49acf685-8a66-4d5a-9277-f881ccc75343\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:57:58.935409 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.935372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49acf685-8a66-4d5a-9277-f881ccc75343-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg\" (UID: \"49acf685-8a66-4d5a-9277-f881ccc75343\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:57:58.935604 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.935452    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w6p8\" (UniqueName: \"kubernetes.io/projected/49acf685-8a66-4d5a-9277-f881ccc75343-kube-api-access-6w6p8\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg\" (UID: \"49acf685-8a66-4d5a-9277-f881ccc75343\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:57:58.937967 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.937950    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49acf685-8a66-4d5a-9277-f881ccc75343-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg\" (UID: \"49acf685-8a66-4d5a-9277-f881ccc75343\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:57:58.943444 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:58.943419    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w6p8\" (UniqueName: \"kubernetes.io/projected/49acf685-8a66-4d5a-9277-f881ccc75343-kube-api-access-6w6p8\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg\" (UID: \"49acf685-8a66-4d5a-9277-f881ccc75343\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:57:59.088318 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:59.088225    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:57:59.219389 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:59.219323    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"]
Jul 28 02:57:59.221760 ip-10-0-140-37 kubenswrapper[2556]: W0728 02:57:59.221734    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49acf685_8a66_4d5a_9277_f881ccc75343.slice/crio-7014ae89e39a48d41a5bd63f154bec761d7bf6012d129372059fe6b40a5c4f9f WatchSource:0}: Error finding container 7014ae89e39a48d41a5bd63f154bec761d7bf6012d129372059fe6b40a5c4f9f: Status 404 returned error can't find the container with id 7014ae89e39a48d41a5bd63f154bec761d7bf6012d129372059fe6b40a5c4f9f
Jul 28 02:57:59.942005 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:59.941969    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg" event={"ID":"49acf685-8a66-4d5a-9277-f881ccc75343","Type":"ContainerStarted","Data":"bc39f0d0187c32b8d9670db2e7237298fe284ee714bc4558862caa8298f17192"}
Jul 28 02:57:59.942420 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:59.942087    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:57:59.942420 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:59.942117    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg" event={"ID":"49acf685-8a66-4d5a-9277-f881ccc75343","Type":"ContainerStarted","Data":"7014ae89e39a48d41a5bd63f154bec761d7bf6012d129372059fe6b40a5c4f9f"}
Jul 28 02:57:59.958929 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:57:59.958882    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg" podStartSLOduration=1.958868378 podStartE2EDuration="1.958868378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:57:59.957480404 +0000 UTC m=+5560.130225671" watchObservedRunningTime="2026-07-28 02:57:59.958868378 +0000 UTC m=+5560.131613628"
Jul 28 02:58:30.948670 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:58:30.948640    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 02:58:35.679618 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:58:35.679582    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:58:56.955989 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:58:56.955953    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 02:59:20.746415 ip-10-0-140-37 kubenswrapper[2556]: E0728 02:59:20.746375    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 02:59:21.302315 ip-10-0-140-37 kubenswrapper[2556]: I0728 02:59:21.302285    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 03:00:20.721946 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:20.721920    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 03:00:20.734428 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:20.734407    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 03:00:35.430410 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:35.430378    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"]
Jul 28 03:00:35.459835 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:35.459806    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:00:35.521782 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:35.521749    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:00:40.948299 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:40.948256    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg" podUID="49acf685-8a66-4d5a-9277-f881ccc75343" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 03:00:50.450539 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.450478    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg" podUID="49acf685-8a66-4d5a-9277-f881ccc75343" containerName="main" containerID="cri-o://bc39f0d0187c32b8d9670db2e7237298fe284ee714bc4558862caa8298f17192" gracePeriod=45
Jul 28 03:00:50.697103 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.697068    2556 generic.go:356] "Generic (PLEG): container finished" podID="49acf685-8a66-4d5a-9277-f881ccc75343" containerID="bc39f0d0187c32b8d9670db2e7237298fe284ee714bc4558862caa8298f17192" exitCode=0
Jul 28 03:00:50.697222 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.697145    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg" event={"ID":"49acf685-8a66-4d5a-9277-f881ccc75343","Type":"ContainerDied","Data":"bc39f0d0187c32b8d9670db2e7237298fe284ee714bc4558862caa8298f17192"}
Jul 28 03:00:50.697222 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.697212    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg" event={"ID":"49acf685-8a66-4d5a-9277-f881ccc75343","Type":"ContainerDied","Data":"7014ae89e39a48d41a5bd63f154bec761d7bf6012d129372059fe6b40a5c4f9f"}
Jul 28 03:00:50.697296 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.697234    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7014ae89e39a48d41a5bd63f154bec761d7bf6012d129372059fe6b40a5c4f9f"
Jul 28 03:00:50.710130 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.710076    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 03:00:50.843901 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.843868    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49acf685-8a66-4d5a-9277-f881ccc75343-tls-certs\") pod \"49acf685-8a66-4d5a-9277-f881ccc75343\" (UID: \"49acf685-8a66-4d5a-9277-f881ccc75343\") "
Jul 28 03:00:50.844092 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.843988    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w6p8\" (UniqueName: \"kubernetes.io/projected/49acf685-8a66-4d5a-9277-f881ccc75343-kube-api-access-6w6p8\") pod \"49acf685-8a66-4d5a-9277-f881ccc75343\" (UID: \"49acf685-8a66-4d5a-9277-f881ccc75343\") "
Jul 28 03:00:50.846424 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.846395    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49acf685-8a66-4d5a-9277-f881ccc75343-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49acf685-8a66-4d5a-9277-f881ccc75343" (UID: "49acf685-8a66-4d5a-9277-f881ccc75343"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 03:00:50.846514 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.846437    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49acf685-8a66-4d5a-9277-f881ccc75343-kube-api-access-6w6p8" (OuterVolumeSpecName: "kube-api-access-6w6p8") pod "49acf685-8a66-4d5a-9277-f881ccc75343" (UID: "49acf685-8a66-4d5a-9277-f881ccc75343"). InnerVolumeSpecName "kube-api-access-6w6p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:00:50.945476 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.945448    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49acf685-8a66-4d5a-9277-f881ccc75343-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:00:50.945476 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:50.945474    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w6p8\" (UniqueName: \"kubernetes.io/projected/49acf685-8a66-4d5a-9277-f881ccc75343-kube-api-access-6w6p8\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:00:51.701777 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:51.701743    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"
Jul 28 03:00:51.721735 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:51.721708    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"]
Jul 28 03:00:51.725738 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:51.725715    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-t5jkg"]
Jul 28 03:00:52.286378 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:52.286344    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49acf685-8a66-4d5a-9277-f881ccc75343" path="/var/lib/kubelet/pods/49acf685-8a66-4d5a-9277-f881ccc75343/volumes"
Jul 28 03:00:52.898359 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:52.898326    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:00:52.997467 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:52.997418    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:00:53.097385 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:53.097352    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:00:59.158171 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:00:59.158127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:01:04.940424 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:04.940386    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:01:05.041689 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:05.041656    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:01:12.758785 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:12.758690    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:01:13.009113 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.009034    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"]
Jul 28 03:01:13.009409 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.009397    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49acf685-8a66-4d5a-9277-f881ccc75343" containerName="main"
Jul 28 03:01:13.009452 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.009411    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="49acf685-8a66-4d5a-9277-f881ccc75343" containerName="main"
Jul 28 03:01:13.009487 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.009466    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="49acf685-8a66-4d5a-9277-f881ccc75343" containerName="main"
Jul 28 03:01:13.012516 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.012498    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:13.016321 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.016295    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 28 03:01:13.016321 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.016307    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-7q545\""
Jul 28 03:01:13.016494 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.016294    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-9s85p\""
Jul 28 03:01:13.016494 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.016294    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 28 03:01:13.016494 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.016333    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 28 03:01:13.022590 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.022548    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"]
Jul 28 03:01:13.030632 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.030612    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c\" (UID: \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:13.030741 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.030676    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpf4t\" (UniqueName: \"kubernetes.io/projected/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-kube-api-access-jpf4t\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c\" (UID: \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:13.131483 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.131454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c\" (UID: \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:13.131625 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.131513    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpf4t\" (UniqueName: \"kubernetes.io/projected/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-kube-api-access-jpf4t\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c\" (UID: \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:13.134122 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.134105    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c\" (UID: \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:13.139731 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.139713    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpf4t\" (UniqueName: \"kubernetes.io/projected/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-kube-api-access-jpf4t\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c\" (UID: \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:13.324201 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.324173    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:13.455620 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.455591    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"]
Jul 28 03:01:13.457732 ip-10-0-140-37 kubenswrapper[2556]: W0728 03:01:13.457704    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfc7a31_eb58_4e26_8cdf_ea72dab52361.slice/crio-06521ea1cab22c258b818767a8b8631e3f47d3143b18bcca40878f14fe8d41cc WatchSource:0}: Error finding container 06521ea1cab22c258b818767a8b8631e3f47d3143b18bcca40878f14fe8d41cc: Status 404 returned error can't find the container with id 06521ea1cab22c258b818767a8b8631e3f47d3143b18bcca40878f14fe8d41cc
Jul 28 03:01:13.801958 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.801870    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c" event={"ID":"fdfc7a31-eb58-4e26-8cdf-ea72dab52361","Type":"ContainerStarted","Data":"31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc"}
Jul 28 03:01:13.801958 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.801928    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c" event={"ID":"fdfc7a31-eb58-4e26-8cdf-ea72dab52361","Type":"ContainerStarted","Data":"06521ea1cab22c258b818767a8b8631e3f47d3143b18bcca40878f14fe8d41cc"}
Jul 28 03:01:13.802373 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.801973    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:13.818734 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:13.818685    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c" podStartSLOduration=1.818669521 podStartE2EDuration="1.818669521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 03:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:01:13.817479865 +0000 UTC m=+5753.990225117" watchObservedRunningTime="2026-07-28 03:01:13.818669521 +0000 UTC m=+5753.991414769"
Jul 28 03:01:14.917159 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:14.917126    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:01:24.302989 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:01:24.302953    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 03:01:24.851386 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:24.851354    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 03:01:38.789866 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:38.789817    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:01:44.808457 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:44.808424    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:01:48.392053 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:48.392019    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:01:48.456162 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:48.456126    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:01:57.265262 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:01:57.265229    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:02:25.344655 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:25.344622    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"]
Jul 28 03:02:25.354050 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:25.354024    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:02:26.613596 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:26.613529    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:02:34.808945 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:34.808907    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c" podUID="fdfc7a31-eb58-4e26-8cdf-ea72dab52361" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 03:02:40.365825 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:40.365787    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c" podUID="fdfc7a31-eb58-4e26-8cdf-ea72dab52361" containerName="main" containerID="cri-o://31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc" gracePeriod=45
Jul 28 03:02:40.613961 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:40.613932    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:02:40.712089 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:40.712011    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpf4t\" (UniqueName: \"kubernetes.io/projected/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-kube-api-access-jpf4t\") pod \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\" (UID: \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\") "
Jul 28 03:02:40.712089 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:40.712063    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-tls-certs\") pod \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\" (UID: \"fdfc7a31-eb58-4e26-8cdf-ea72dab52361\") "
Jul 28 03:02:40.714453 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:40.714426    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-kube-api-access-jpf4t" (OuterVolumeSpecName: "kube-api-access-jpf4t") pod "fdfc7a31-eb58-4e26-8cdf-ea72dab52361" (UID: "fdfc7a31-eb58-4e26-8cdf-ea72dab52361"). InnerVolumeSpecName "kube-api-access-jpf4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:02:40.714613 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:40.714464    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdfc7a31-eb58-4e26-8cdf-ea72dab52361" (UID: "fdfc7a31-eb58-4e26-8cdf-ea72dab52361"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 03:02:40.812884 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:40.812856    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpf4t\" (UniqueName: \"kubernetes.io/projected/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-kube-api-access-jpf4t\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:02:40.812884 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:40.812881    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfc7a31-eb58-4e26-8cdf-ea72dab52361-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:02:41.196495 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.196457    2556 generic.go:356] "Generic (PLEG): container finished" podID="fdfc7a31-eb58-4e26-8cdf-ea72dab52361" containerID="31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc" exitCode=0
Jul 28 03:02:41.196707 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.196503    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c" event={"ID":"fdfc7a31-eb58-4e26-8cdf-ea72dab52361","Type":"ContainerDied","Data":"31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc"}
Jul 28 03:02:41.196707 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.196543    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c" event={"ID":"fdfc7a31-eb58-4e26-8cdf-ea72dab52361","Type":"ContainerDied","Data":"06521ea1cab22c258b818767a8b8631e3f47d3143b18bcca40878f14fe8d41cc"}
Jul 28 03:02:41.196707 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.196505    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"
Jul 28 03:02:41.196707 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.196583    2556 scope.go:117] "RemoveContainer" containerID="31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc"
Jul 28 03:02:41.205594 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.205578    2556 scope.go:117] "RemoveContainer" containerID="31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc"
Jul 28 03:02:41.205828 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:02:41.205813    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc\": container with ID starting with 31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc not found: ID does not exist" containerID="31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc"
Jul 28 03:02:41.205877 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.205836    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc"} err="failed to get container status \"31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc\": rpc error: code = NotFound desc = could not find container \"31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc\": container with ID starting with 31652d155ce1a10749b4cf034f861c9b30841ea74150c81148da00df55a29acc not found: ID does not exist"
Jul 28 03:02:41.216857 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.216834    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"]
Jul 28 03:02:41.220519 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.220497    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-fgn8c"]
Jul 28 03:02:41.844573 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.844538    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:02:41.944423 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:41.944387    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:02:42.044860 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:42.044821    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:02:42.286986 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:42.286908    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdfc7a31-eb58-4e26-8cdf-ea72dab52361" path="/var/lib/kubelet/pods/fdfc7a31-eb58-4e26-8cdf-ea72dab52361/volumes"
Jul 28 03:02:44.486773 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.486741    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"]
Jul 28 03:02:44.487152 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.487091    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfc7a31-eb58-4e26-8cdf-ea72dab52361" containerName="main"
Jul 28 03:02:44.487152 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.487102    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfc7a31-eb58-4e26-8cdf-ea72dab52361" containerName="main"
Jul 28 03:02:44.487230 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.487157    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfc7a31-eb58-4e26-8cdf-ea72dab52361" containerName="main"
Jul 28 03:02:44.491437 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.491419    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:02:44.495276 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.495257    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 03:02:44.495376 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.495286    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 03:02:44.495376 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.495286    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-pmsft\""
Jul 28 03:02:44.495499 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.495380    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 03:02:44.497917 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.497893    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"]
Jul 28 03:02:44.649649 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.649614    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f797e525-5eb0-46fb-8823-eb578ad552bf-cert\") pod \"llmisvc-controller-manager-6fcdf65d46-s9w4q\" (UID: \"f797e525-5eb0-46fb-8823-eb578ad552bf\") " pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:02:44.649833 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.649670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5rl\" (UniqueName: \"kubernetes.io/projected/f797e525-5eb0-46fb-8823-eb578ad552bf-kube-api-access-6g5rl\") pod \"llmisvc-controller-manager-6fcdf65d46-s9w4q\" (UID: \"f797e525-5eb0-46fb-8823-eb578ad552bf\") " pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:02:44.751185 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.751092    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f797e525-5eb0-46fb-8823-eb578ad552bf-cert\") pod \"llmisvc-controller-manager-6fcdf65d46-s9w4q\" (UID: \"f797e525-5eb0-46fb-8823-eb578ad552bf\") " pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:02:44.751332 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.751188    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5rl\" (UniqueName: \"kubernetes.io/projected/f797e525-5eb0-46fb-8823-eb578ad552bf-kube-api-access-6g5rl\") pod \"llmisvc-controller-manager-6fcdf65d46-s9w4q\" (UID: \"f797e525-5eb0-46fb-8823-eb578ad552bf\") " pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:02:44.753721 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.753698    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f797e525-5eb0-46fb-8823-eb578ad552bf-cert\") pod \"llmisvc-controller-manager-6fcdf65d46-s9w4q\" (UID: \"f797e525-5eb0-46fb-8823-eb578ad552bf\") " pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:02:44.760388 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.760363    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5rl\" (UniqueName: \"kubernetes.io/projected/f797e525-5eb0-46fb-8823-eb578ad552bf-kube-api-access-6g5rl\") pod \"llmisvc-controller-manager-6fcdf65d46-s9w4q\" (UID: \"f797e525-5eb0-46fb-8823-eb578ad552bf\") " pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:02:44.802202 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.802175    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:02:44.932319 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.932296    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"]
Jul 28 03:02:44.934180 ip-10-0-140-37 kubenswrapper[2556]: W0728 03:02:44.934152    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf797e525_5eb0_46fb_8823_eb578ad552bf.slice/crio-bd78c042168095b9168c302ec53ed1704b5dbcfc6d4d23963267c77b1ddd060f WatchSource:0}: Error finding container bd78c042168095b9168c302ec53ed1704b5dbcfc6d4d23963267c77b1ddd060f: Status 404 returned error can't find the container with id bd78c042168095b9168c302ec53ed1704b5dbcfc6d4d23963267c77b1ddd060f
Jul 28 03:02:44.935851 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:44.935837    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 03:02:45.218242 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:45.218206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q" event={"ID":"f797e525-5eb0-46fb-8823-eb578ad552bf","Type":"ContainerStarted","Data":"bd78c042168095b9168c302ec53ed1704b5dbcfc6d4d23963267c77b1ddd060f"}
Jul 28 03:02:47.663159 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:47.663124    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:02:49.237871 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:49.237831    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q" event={"ID":"f797e525-5eb0-46fb-8823-eb578ad552bf","Type":"ContainerStarted","Data":"81062156bccbf793ad8e89181b125bdf60188a8fd76ba4d1eef1fb1f12e5e9df"}
Jul 28 03:02:49.238316 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:49.237900    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:02:49.255032 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:49.254983    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q" podStartSLOduration=1.9058779129999999 podStartE2EDuration="5.25496734s" totalImagesPullingTime="3.349089427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 03:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:02:49.252988303 +0000 UTC m=+5849.425733556" watchObservedRunningTime="2026-07-28 03:02:49.25496734 +0000 UTC m=+5849.427712588"
Jul 28 03:02:53.218299 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:53.218268    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:02:53.283063 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:02:53.283036    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:03:20.245162 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:03:20.245133    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6fcdf65d46-s9w4q"
Jul 28 03:03:27.852705 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:03:27.852650    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 03:03:28.408075 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:03:28.408044    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 03:03:58.925975 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:03:58.925941    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:04:01.241783 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:04:01.241752    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:04:07.082866 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:04:07.082835    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:04:07.144407 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:04:07.144363    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:04:21.057033 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:04:21.056956    2556 scope.go:117] "RemoveContainer" containerID="bc39f0d0187c32b8d9670db2e7237298fe284ee714bc4558862caa8298f17192"
Jul 28 03:05:20.750466 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:05:20.750434    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 03:05:20.763177 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:05:20.763156    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wrg24_69a3339f-758b-4280-98bc-1bde23005e7b/ovn-acl-logging/0.log"
Jul 28 03:05:31.408920 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:05:31.408878    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 03:05:31.964956 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:05:31.964924    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 03:07:34.965897 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:07:34.965805    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc" podUID="568c4ac2-0da4-4b97-8d6f-f9554a808bbb"
Jul 28 03:07:35.531915 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:35.531886    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc"
Jul 28 03:07:38.615648 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:38.615614    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:07:45.085090 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.085053    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"]
Jul 28 03:07:45.099959 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.099909    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:07:45.105987 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.105959    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"]
Jul 28 03:07:45.116042 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:07:45.116011    2556 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 03:07:45.116042 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 03:07:45.116042 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 03:07:45.116042 ip-10-0-140-37 kubenswrapper[2556]:  > containerID="75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4" cmd=["/bin/sleep","15"]
Jul 28 03:07:45.116339 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.116054    2556 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8" message="" err=<
Jul 28 03:07:45.116339 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 03:07:45.116339 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 03:07:45.116339 ip-10-0-140-37 kubenswrapper[2556]:  >
Jul 28 03:07:45.116339 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:07:45.116067    2556 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 03:07:45.116339 ip-10-0-140-37 kubenswrapper[2556]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 03:07:45.116339 ip-10-0-140-37 kubenswrapper[2556]:         , stdout: , stderr: , exit code -1
Jul 28 03:07:45.116339 ip-10-0-140-37 kubenswrapper[2556]:  > pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8" podUID="febc0b4d-1902-4a04-853d-b525b0d87a47" containerName="main" containerID="cri-o://75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4"
Jul 28 03:07:45.116339 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.116099    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8" podUID="febc0b4d-1902-4a04-853d-b525b0d87a47" containerName="main" containerID="cri-o://75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4" gracePeriod=60
Jul 28 03:07:45.157893 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.157864    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:07:45.460507 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.460483    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 03:07:45.546492 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.546467    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-home\") pod \"febc0b4d-1902-4a04-853d-b525b0d87a47\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") "
Jul 28 03:07:45.546681 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.546511    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febc0b4d-1902-4a04-853d-b525b0d87a47-tls-certs\") pod \"febc0b4d-1902-4a04-853d-b525b0d87a47\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") "
Jul 28 03:07:45.546681 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.546540    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hwm4\" (UniqueName: \"kubernetes.io/projected/febc0b4d-1902-4a04-853d-b525b0d87a47-kube-api-access-6hwm4\") pod \"febc0b4d-1902-4a04-853d-b525b0d87a47\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") "
Jul 28 03:07:45.546681 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.546573    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-tmp-dir\") pod \"febc0b4d-1902-4a04-853d-b525b0d87a47\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") "
Jul 28 03:07:45.546681 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.546606    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-dshm\") pod \"febc0b4d-1902-4a04-853d-b525b0d87a47\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") "
Jul 28 03:07:45.546681 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.546656    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-model-cache\") pod \"febc0b4d-1902-4a04-853d-b525b0d87a47\" (UID: \"febc0b4d-1902-4a04-853d-b525b0d87a47\") "
Jul 28 03:07:45.546973 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.546776    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-home" (OuterVolumeSpecName: "home") pod "febc0b4d-1902-4a04-853d-b525b0d87a47" (UID: "febc0b4d-1902-4a04-853d-b525b0d87a47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:07:45.546973 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.546867    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "febc0b4d-1902-4a04-853d-b525b0d87a47" (UID: "febc0b4d-1902-4a04-853d-b525b0d87a47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:07:45.546973 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.546957    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-model-cache" (OuterVolumeSpecName: "model-cache") pod "febc0b4d-1902-4a04-853d-b525b0d87a47" (UID: "febc0b4d-1902-4a04-853d-b525b0d87a47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:07:45.547125 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.547004    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-tmp-dir\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:07:45.547125 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.547016    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-home\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:07:45.548932 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.548909    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febc0b4d-1902-4a04-853d-b525b0d87a47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "febc0b4d-1902-4a04-853d-b525b0d87a47" (UID: "febc0b4d-1902-4a04-853d-b525b0d87a47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 03:07:45.548932 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.548908    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-dshm" (OuterVolumeSpecName: "dshm") pod "febc0b4d-1902-4a04-853d-b525b0d87a47" (UID: "febc0b4d-1902-4a04-853d-b525b0d87a47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:07:45.549093 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.549018    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febc0b4d-1902-4a04-853d-b525b0d87a47-kube-api-access-6hwm4" (OuterVolumeSpecName: "kube-api-access-6hwm4") pod "febc0b4d-1902-4a04-853d-b525b0d87a47" (UID: "febc0b4d-1902-4a04-853d-b525b0d87a47"). InnerVolumeSpecName "kube-api-access-6hwm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:07:45.585705 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.585681    2556 generic.go:356] "Generic (PLEG): container finished" podID="febc0b4d-1902-4a04-853d-b525b0d87a47" containerID="75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4" exitCode=0
Jul 28 03:07:45.585817 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.585726    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"
Jul 28 03:07:45.585817 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.585773    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8" event={"ID":"febc0b4d-1902-4a04-853d-b525b0d87a47","Type":"ContainerDied","Data":"75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4"}
Jul 28 03:07:45.585943 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.585829    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8" event={"ID":"febc0b4d-1902-4a04-853d-b525b0d87a47","Type":"ContainerDied","Data":"87e7c8697a3ca2ea9c1f727183a4661bd0150e1d0cd0ccad55005501a70507b5"}
Jul 28 03:07:45.585943 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.585864    2556 scope.go:117] "RemoveContainer" containerID="75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4"
Jul 28 03:07:45.595770 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.595755    2556 scope.go:117] "RemoveContainer" containerID="75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4"
Jul 28 03:07:45.595992 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:07:45.595977    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4\": container with ID starting with 75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4 not found: ID does not exist" containerID="75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4"
Jul 28 03:07:45.596041 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.595999    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4"} err="failed to get container status \"75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4\": rpc error: code = NotFound desc = could not find container \"75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4\": container with ID starting with 75952d6d5106b509fef1a5a621514f0fee5da5cb0bb70885ccfb1a32f4c057e4 not found: ID does not exist"
Jul 28 03:07:45.606773 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.606752    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"]
Jul 28 03:07:45.612943 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.612915    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8"]
Jul 28 03:07:45.647682 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.647615    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-dshm\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:07:45.647682 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.647639    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febc0b4d-1902-4a04-853d-b525b0d87a47-model-cache\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:07:45.647682 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.647653    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febc0b4d-1902-4a04-853d-b525b0d87a47-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:07:45.647682 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:45.647664    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hwm4\" (UniqueName: \"kubernetes.io/projected/febc0b4d-1902-4a04-853d-b525b0d87a47-kube-api-access-6hwm4\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:07:46.286317 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:46.286282    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="febc0b4d-1902-4a04-853d-b525b0d87a47" path="/var/lib/kubelet/pods/febc0b4d-1902-4a04-853d-b525b0d87a47/volumes"
Jul 28 03:07:53.622266 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:07:53.622221    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq" podUID="2fe510b7-a916-4b71-81dc-7e23ae6e3159" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 03:08:00.107012 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.106944    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq" podUID="2fe510b7-a916-4b71-81dc-7e23ae6e3159" containerName="main" containerID="cri-o://3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37" gracePeriod=45
Jul 28 03:08:00.355544 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.355515    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 03:08:00.474579 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.474479    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkx4z\" (UniqueName: \"kubernetes.io/projected/2fe510b7-a916-4b71-81dc-7e23ae6e3159-kube-api-access-hkx4z\") pod \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\" (UID: \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\") "
Jul 28 03:08:00.474579 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.474520    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe510b7-a916-4b71-81dc-7e23ae6e3159-tls-certs\") pod \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\" (UID: \"2fe510b7-a916-4b71-81dc-7e23ae6e3159\") "
Jul 28 03:08:00.476831 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.476798    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe510b7-a916-4b71-81dc-7e23ae6e3159-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fe510b7-a916-4b71-81dc-7e23ae6e3159" (UID: "2fe510b7-a916-4b71-81dc-7e23ae6e3159"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 03:08:00.476950 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.476899    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe510b7-a916-4b71-81dc-7e23ae6e3159-kube-api-access-hkx4z" (OuterVolumeSpecName: "kube-api-access-hkx4z") pod "2fe510b7-a916-4b71-81dc-7e23ae6e3159" (UID: "2fe510b7-a916-4b71-81dc-7e23ae6e3159"). InnerVolumeSpecName "kube-api-access-hkx4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:08:00.576165 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.576131    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkx4z\" (UniqueName: \"kubernetes.io/projected/2fe510b7-a916-4b71-81dc-7e23ae6e3159-kube-api-access-hkx4z\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:08:00.576165 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.576161    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe510b7-a916-4b71-81dc-7e23ae6e3159-tls-certs\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:08:00.656551 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.656518    2556 generic.go:356] "Generic (PLEG): container finished" podID="2fe510b7-a916-4b71-81dc-7e23ae6e3159" containerID="3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37" exitCode=0
Jul 28 03:08:00.656551 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.656570    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq" event={"ID":"2fe510b7-a916-4b71-81dc-7e23ae6e3159","Type":"ContainerDied","Data":"3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37"}
Jul 28 03:08:00.656781 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.656582    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"
Jul 28 03:08:00.656781 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.656614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq" event={"ID":"2fe510b7-a916-4b71-81dc-7e23ae6e3159","Type":"ContainerDied","Data":"f2dea12d4e7eeb67d234dd14401f9fad1acebb2727d8f3042369f960ccf49410"}
Jul 28 03:08:00.656781 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.656636    2556 scope.go:117] "RemoveContainer" containerID="3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37"
Jul 28 03:08:00.665616 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.665597    2556 scope.go:117] "RemoveContainer" containerID="3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37"
Jul 28 03:08:00.665854 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:08:00.665834    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37\": container with ID starting with 3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37 not found: ID does not exist" containerID="3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37"
Jul 28 03:08:00.665917 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.665862    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37"} err="failed to get container status \"3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37\": rpc error: code = NotFound desc = could not find container \"3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37\": container with ID starting with 3c310f2c389377b517f6096c4aad94cdb45f659c6c75dc701b6130b1c044eb37 not found: ID does not exist"
Jul 28 03:08:00.678230 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.678206    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"]
Jul 28 03:08:00.685790 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:00.685767    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq"]
Jul 28 03:08:02.286213 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:02.286178    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fe510b7-a916-4b71-81dc-7e23ae6e3159" path="/var/lib/kubelet/pods/2fe510b7-a916-4b71-81dc-7e23ae6e3159/volumes"
Jul 28 03:08:06.949049 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.949004    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n226x/must-gather-cgszc"]
Jul 28 03:08:06.949603 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.949329    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="febc0b4d-1902-4a04-853d-b525b0d87a47" containerName="main"
Jul 28 03:08:06.949603 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.949340    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="febc0b4d-1902-4a04-853d-b525b0d87a47" containerName="main"
Jul 28 03:08:06.949603 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.949348    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe510b7-a916-4b71-81dc-7e23ae6e3159" containerName="main"
Jul 28 03:08:06.949603 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.949353    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe510b7-a916-4b71-81dc-7e23ae6e3159" containerName="main"
Jul 28 03:08:06.949603 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.949423    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="febc0b4d-1902-4a04-853d-b525b0d87a47" containerName="main"
Jul 28 03:08:06.949603 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.949431    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fe510b7-a916-4b71-81dc-7e23ae6e3159" containerName="main"
Jul 28 03:08:06.954128 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.954108    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:08:06.956686 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.956665    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n226x\"/\"openshift-service-ca.crt\""
Jul 28 03:08:06.956791 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.956702    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n226x\"/\"kube-root-ca.crt\""
Jul 28 03:08:06.957851 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.957718    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n226x\"/\"default-dockercfg-c57br\""
Jul 28 03:08:06.959577 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.959543    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n226x/must-gather-cgszc"]
Jul 28 03:08:06.964686 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:06.964661    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:08:07.030623 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.030588    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:08:07.031750 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.031733    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fd76eea0-7751-4d83-982a-acddc0556d53-must-gather-output\") pod \"must-gather-cgszc\" (UID: \"fd76eea0-7751-4d83-982a-acddc0556d53\") " pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:08:07.031818 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.031790    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm9v5\" (UniqueName: \"kubernetes.io/projected/fd76eea0-7751-4d83-982a-acddc0556d53-kube-api-access-qm9v5\") pod \"must-gather-cgszc\" (UID: \"fd76eea0-7751-4d83-982a-acddc0556d53\") " pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:08:07.132711 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.132681    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm9v5\" (UniqueName: \"kubernetes.io/projected/fd76eea0-7751-4d83-982a-acddc0556d53-kube-api-access-qm9v5\") pod \"must-gather-cgszc\" (UID: \"fd76eea0-7751-4d83-982a-acddc0556d53\") " pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:08:07.132863 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.132740    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fd76eea0-7751-4d83-982a-acddc0556d53-must-gather-output\") pod \"must-gather-cgszc\" (UID: \"fd76eea0-7751-4d83-982a-acddc0556d53\") " pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:08:07.133081 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.133063    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fd76eea0-7751-4d83-982a-acddc0556d53-must-gather-output\") pod \"must-gather-cgszc\" (UID: \"fd76eea0-7751-4d83-982a-acddc0556d53\") " pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:08:07.140957 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.140931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm9v5\" (UniqueName: \"kubernetes.io/projected/fd76eea0-7751-4d83-982a-acddc0556d53-kube-api-access-qm9v5\") pod \"must-gather-cgszc\" (UID: \"fd76eea0-7751-4d83-982a-acddc0556d53\") " pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:08:07.264398 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.264322    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:08:07.391774 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.391750    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n226x/must-gather-cgszc"]
Jul 28 03:08:07.394343 ip-10-0-140-37 kubenswrapper[2556]: W0728 03:08:07.394317    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd76eea0_7751_4d83_982a_acddc0556d53.slice/crio-adde49607b99598622f6d0f9ad3374961620e4ec3014c83d8744703a83054ad7 WatchSource:0}: Error finding container adde49607b99598622f6d0f9ad3374961620e4ec3014c83d8744703a83054ad7: Status 404 returned error can't find the container with id adde49607b99598622f6d0f9ad3374961620e4ec3014c83d8744703a83054ad7
Jul 28 03:08:07.396159 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.396141    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 03:08:07.692172 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:07.692140    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n226x/must-gather-cgszc" event={"ID":"fd76eea0-7751-4d83-982a-acddc0556d53","Type":"ContainerStarted","Data":"adde49607b99598622f6d0f9ad3374961620e4ec3014c83d8744703a83054ad7"}
Jul 28 03:08:14.735178 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:14.735142    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n226x/must-gather-cgszc" event={"ID":"fd76eea0-7751-4d83-982a-acddc0556d53","Type":"ContainerStarted","Data":"9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9"}
Jul 28 03:08:14.735642 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:14.735195    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n226x/must-gather-cgszc" event={"ID":"fd76eea0-7751-4d83-982a-acddc0556d53","Type":"ContainerStarted","Data":"8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd"}
Jul 28 03:08:14.752634 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:14.752573    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n226x/must-gather-cgszc" podStartSLOduration=2.245580166 podStartE2EDuration="8.752536966s" totalImagesPullingTime="6.5069568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 03:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:08:14.752075946 +0000 UTC m=+6174.924821193" watchObservedRunningTime="2026-07-28 03:08:14.752536966 +0000 UTC m=+6174.925282216"
Jul 28 03:08:33.912348 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:33.912318    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:33.943116 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:33.943077    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:35.255854 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:35.255822    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:35.288715 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:35.288658    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:36.407028 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:36.406999    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:36.421740 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:36.421711    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:37.485803 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:37.485767    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:37.502319 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:37.502295    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:38.544812 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:38.544782    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:38.559719 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:38.559698    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:39.652452 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:39.652417    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:39.667571 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:39.667520    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:40.741432 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:40.741400    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:40.757794 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:40.757767    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:41.801217 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:41.801139    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:41.815485 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:41.815466    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:42.871071 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:42.871042    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:42.890684 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:42.890661    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:43.972031 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:43.972000    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:43.986138 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:43.986110    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:45.087379 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:45.087349    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:45.102602 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:45.102550    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:46.163108 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:46.163082    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:46.178237 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:46.178193    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:47.284651 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:47.284625    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:47.300251 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:47.300228    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:48.398871 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:48.398842    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:48.414248 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:48.414220    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:49.478434 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:49.478406    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:49.497270 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:49.497220    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:50.582633 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:50.582600    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:50.602039 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:50.602015    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:51.668420 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:51.668387    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:51.683148 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:51.683122    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:52.747198 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:52.747169    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:52.767124 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:52.767099    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:53.809345 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:53.809314    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:53.824200 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:53.824167    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:54.881808 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:54.881775    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:54.897399 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:54.897359    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:55.959511 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:55.959486    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:55.974007 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:55.973979    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:57.030857 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:57.030828    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:57.045178 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:57.045149    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:58.101350 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:58.101321    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:58.121683 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:58.121651    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:08:59.208950 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:59.208919    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:08:59.233503 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:08:59.233471    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:09:00.316975 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:00.316946    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:09:00.334220 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:00.334197    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:09:00.974110 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:00.974079    2556 generic.go:356] "Generic (PLEG): container finished" podID="fd76eea0-7751-4d83-982a-acddc0556d53" containerID="8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd" exitCode=0
Jul 28 03:09:00.974316 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:00.974139    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n226x/must-gather-cgszc" event={"ID":"fd76eea0-7751-4d83-982a-acddc0556d53","Type":"ContainerDied","Data":"8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd"}
Jul 28 03:09:00.974513 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:00.974494    2556 scope.go:117] "RemoveContainer" containerID="8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd"
Jul 28 03:09:01.816592 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:01.816549    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-n226x_must-gather-cgszc_fd76eea0-7751-4d83-982a-acddc0556d53/gather/0.log"
Jul 28 03:09:02.264691 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.264653    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:09:02.397688 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.397652    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6q4dh/must-gather-ng2z5"]
Jul 28 03:09:02.400880 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.400864    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q4dh/must-gather-ng2z5"
Jul 28 03:09:02.403607 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.403585    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q4dh\"/\"openshift-service-ca.crt\""
Jul 28 03:09:02.403720 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.403585    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q4dh\"/\"kube-root-ca.crt\""
Jul 28 03:09:02.410254 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.410232    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q4dh/must-gather-ng2z5"]
Jul 28 03:09:02.414586 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.414551    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:09:02.449431 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.449403    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d075d98a-ecd9-41b2-9307-c864ec8da9b4-must-gather-output\") pod \"must-gather-ng2z5\" (UID: \"d075d98a-ecd9-41b2-9307-c864ec8da9b4\") " pod="openshift-must-gather-6q4dh/must-gather-ng2z5"
Jul 28 03:09:02.449607 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.449542    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzh7\" (UniqueName: \"kubernetes.io/projected/d075d98a-ecd9-41b2-9307-c864ec8da9b4-kube-api-access-hhzh7\") pod \"must-gather-ng2z5\" (UID: \"d075d98a-ecd9-41b2-9307-c864ec8da9b4\") " pod="openshift-must-gather-6q4dh/must-gather-ng2z5"
Jul 28 03:09:02.471421 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.471383    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:09:02.550982 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.550911    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzh7\" (UniqueName: \"kubernetes.io/projected/d075d98a-ecd9-41b2-9307-c864ec8da9b4-kube-api-access-hhzh7\") pod \"must-gather-ng2z5\" (UID: \"d075d98a-ecd9-41b2-9307-c864ec8da9b4\") " pod="openshift-must-gather-6q4dh/must-gather-ng2z5"
Jul 28 03:09:02.550982 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.550967    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d075d98a-ecd9-41b2-9307-c864ec8da9b4-must-gather-output\") pod \"must-gather-ng2z5\" (UID: \"d075d98a-ecd9-41b2-9307-c864ec8da9b4\") " pod="openshift-must-gather-6q4dh/must-gather-ng2z5"
Jul 28 03:09:02.551362 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.551342    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d075d98a-ecd9-41b2-9307-c864ec8da9b4-must-gather-output\") pod \"must-gather-ng2z5\" (UID: \"d075d98a-ecd9-41b2-9307-c864ec8da9b4\") " pod="openshift-must-gather-6q4dh/must-gather-ng2z5"
Jul 28 03:09:02.559327 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.559308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzh7\" (UniqueName: \"kubernetes.io/projected/d075d98a-ecd9-41b2-9307-c864ec8da9b4-kube-api-access-hhzh7\") pod \"must-gather-ng2z5\" (UID: \"d075d98a-ecd9-41b2-9307-c864ec8da9b4\") " pod="openshift-must-gather-6q4dh/must-gather-ng2z5"
Jul 28 03:09:02.710890 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.710853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q4dh/must-gather-ng2z5"
Jul 28 03:09:02.838374 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.838346    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q4dh/must-gather-ng2z5"]
Jul 28 03:09:02.840502 ip-10-0-140-37 kubenswrapper[2556]: W0728 03:09:02.840475    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd075d98a_ecd9_41b2_9307_c864ec8da9b4.slice/crio-41c799406fc8a731018541003b84ccb0ebebda87ab7057c010a83d2f1b495c31 WatchSource:0}: Error finding container 41c799406fc8a731018541003b84ccb0ebebda87ab7057c010a83d2f1b495c31: Status 404 returned error can't find the container with id 41c799406fc8a731018541003b84ccb0ebebda87ab7057c010a83d2f1b495c31
Jul 28 03:09:02.985622 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:02.985588    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q4dh/must-gather-ng2z5" event={"ID":"d075d98a-ecd9-41b2-9307-c864ec8da9b4","Type":"ContainerStarted","Data":"41c799406fc8a731018541003b84ccb0ebebda87ab7057c010a83d2f1b495c31"}
Jul 28 03:09:03.994704 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:03.994660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q4dh/must-gather-ng2z5" event={"ID":"d075d98a-ecd9-41b2-9307-c864ec8da9b4","Type":"ContainerStarted","Data":"845fce7d602adb7da62088b0b6398406e182ce273214e752e7cd554d0dc63399"}
Jul 28 03:09:03.995204 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:03.994718    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q4dh/must-gather-ng2z5" event={"ID":"d075d98a-ecd9-41b2-9307-c864ec8da9b4","Type":"ContainerStarted","Data":"f685fde689fe35012ec25baebc3dff888567dbcc505dcbdd7fa36b8a8a2ea34a"}
Jul 28 03:09:04.011248 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:04.011186    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6q4dh/must-gather-ng2z5" podStartSLOduration=1.357854038 podStartE2EDuration="2.011165979s" totalImagesPullingTime="653.311941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 03:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:09:04.010042346 +0000 UTC m=+6224.182787615" watchObservedRunningTime="2026-07-28 03:09:04.011165979 +0000 UTC m=+6224.183911228"
Jul 28 03:09:04.943105 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:04.943071    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hwr6q_ffc9bda4-ee09-499b-bb30-db6c295d9ec2/global-pull-secret-syncer/0.log"
Jul 28 03:09:05.063338 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:05.063310    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log"
Jul 28 03:09:05.233102 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:05.233026    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-37.ec2.internal_ae809c78f68e15731047d4f8c293c8a9/haproxy/0.log"
Jul 28 03:09:07.244484 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.244437    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-n226x/must-gather-cgszc"]
Jul 28 03:09:07.245063 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.244825    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-n226x/must-gather-cgszc" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" containerName="copy" containerID="cri-o://9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9" gracePeriod=2
Jul 28 03:09:07.247948 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.247906    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-n226x/must-gather-cgszc"]
Jul 28 03:09:07.250155 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:09:07.250115    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cgszc\" is forbidden: User \"system:node:ip-10-0-140-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n226x\": no relationship found between node 'ip-10-0-140-37.ec2.internal' and this object" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:09:07.605011 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.604908    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-n226x_must-gather-cgszc_fd76eea0-7751-4d83-982a-acddc0556d53/copy/0.log"
Jul 28 03:09:07.609982 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.609941    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:09:07.612762 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:09:07.612717    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cgszc\" is forbidden: User \"system:node:ip-10-0-140-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n226x\": no relationship found between node 'ip-10-0-140-37.ec2.internal' and this object" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:09:07.701576 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.701514    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fd76eea0-7751-4d83-982a-acddc0556d53-must-gather-output\") pod \"fd76eea0-7751-4d83-982a-acddc0556d53\" (UID: \"fd76eea0-7751-4d83-982a-acddc0556d53\") "
Jul 28 03:09:07.701786 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.701603    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm9v5\" (UniqueName: \"kubernetes.io/projected/fd76eea0-7751-4d83-982a-acddc0556d53-kube-api-access-qm9v5\") pod \"fd76eea0-7751-4d83-982a-acddc0556d53\" (UID: \"fd76eea0-7751-4d83-982a-acddc0556d53\") "
Jul 28 03:09:07.712396 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.712113    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd76eea0-7751-4d83-982a-acddc0556d53-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "fd76eea0-7751-4d83-982a-acddc0556d53" (UID: "fd76eea0-7751-4d83-982a-acddc0556d53"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:09:07.719625 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.717221    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd76eea0-7751-4d83-982a-acddc0556d53-kube-api-access-qm9v5" (OuterVolumeSpecName: "kube-api-access-qm9v5") pod "fd76eea0-7751-4d83-982a-acddc0556d53" (UID: "fd76eea0-7751-4d83-982a-acddc0556d53"). InnerVolumeSpecName "kube-api-access-qm9v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:09:07.803168 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.803125    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm9v5\" (UniqueName: \"kubernetes.io/projected/fd76eea0-7751-4d83-982a-acddc0556d53-kube-api-access-qm9v5\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:09:07.803168 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:07.803168    2556 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fd76eea0-7751-4d83-982a-acddc0556d53-must-gather-output\") on node \"ip-10-0-140-37.ec2.internal\" DevicePath \"\""
Jul 28 03:09:08.022935 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.022844    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-n226x_must-gather-cgszc_fd76eea0-7751-4d83-982a-acddc0556d53/copy/0.log"
Jul 28 03:09:08.026579 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.023324    2556 generic.go:356] "Generic (PLEG): container finished" podID="fd76eea0-7751-4d83-982a-acddc0556d53" containerID="9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9" exitCode=143
Jul 28 03:09:08.026579 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.023380    2556 scope.go:117] "RemoveContainer" containerID="9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9"
Jul 28 03:09:08.026579 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.023393    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:09:08.040873 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:09:08.040820    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cgszc\" is forbidden: User \"system:node:ip-10-0-140-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n226x\": no relationship found between node 'ip-10-0-140-37.ec2.internal' and this object" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:09:08.045309 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:09:08.045258    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cgszc\" is forbidden: User \"system:node:ip-10-0-140-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n226x\": no relationship found between node 'ip-10-0-140-37.ec2.internal' and this object" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" pod="openshift-must-gather-n226x/must-gather-cgszc"
Jul 28 03:09:08.060113 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.056777    2556 scope.go:117] "RemoveContainer" containerID="8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd"
Jul 28 03:09:08.089010 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.087487    2556 scope.go:117] "RemoveContainer" containerID="9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9"
Jul 28 03:09:08.089010 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:09:08.087892    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9\": container with ID starting with 9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9 not found: ID does not exist" containerID="9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9"
Jul 28 03:09:08.089010 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.087929    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9"} err="failed to get container status \"9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9\": rpc error: code = NotFound desc = could not find container \"9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9\": container with ID starting with 9712b7a96a19f0ecc7d85050a4e57312517674485876a49aab208eb56a171aa9 not found: ID does not exist"
Jul 28 03:09:08.089010 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.087956    2556 scope.go:117] "RemoveContainer" containerID="8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd"
Jul 28 03:09:08.089010 ip-10-0-140-37 kubenswrapper[2556]: E0728 03:09:08.088269    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd\": container with ID starting with 8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd not found: ID does not exist" containerID="8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd"
Jul 28 03:09:08.089010 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.088302    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd"} err="failed to get container status \"8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd\": rpc error: code = NotFound desc = could not find container \"8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd\": container with ID starting with 8852d111397376d852c13390512a42a258f092704b03fbc06a68d96bcb9c69fd not found: ID does not exist"
Jul 28 03:09:08.193951 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.193792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:09:08.291465 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.291390    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" path="/var/lib/kubelet/pods/fd76eea0-7751-4d83-982a-acddc0556d53/volumes"
Jul 28 03:09:08.294049 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:08.294009    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bftn8_2d20181f-51d1-42ae-b14e-411cda629c02/konnectivity-agent/0.log" limit=1024
Jul 28 03:09:10.486957 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:10.486920    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-n86fd_fe32101c-ed53-4867-8572-0d8016433f47/limitador/0.log"
Jul 28 03:09:10.535594 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:10.535540    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-vnhf4_57dc03c7-38f0-4018-b784-75857daddee3/manager/0.log"
Jul 28 03:09:12.073386 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.073339    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xr9dr_60477cfa-b522-48b8-9280-c3528879b2e0/kube-state-metrics/0.log"
Jul 28 03:09:12.092163 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.092115    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xr9dr_60477cfa-b522-48b8-9280-c3528879b2e0/kube-rbac-proxy-main/0.log"
Jul 28 03:09:12.115950 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.115876    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xr9dr_60477cfa-b522-48b8-9280-c3528879b2e0/kube-rbac-proxy-self/0.log"
Jul 28 03:09:12.390400 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.390366    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tfknf_b31ad112-13c2-4435-a57e-008902a2c744/node-exporter/0.log"
Jul 28 03:09:12.415252 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.415224    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tfknf_b31ad112-13c2-4435-a57e-008902a2c744/kube-rbac-proxy/0.log"
Jul 28 03:09:12.443270 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.443239    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tfknf_b31ad112-13c2-4435-a57e-008902a2c744/init-textfile/0.log"
Jul 28 03:09:12.586316 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.586287    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfec3aee-387b-4b53-8e48-760e99709d2e/prometheus/0.log"
Jul 28 03:09:12.606600 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.606547    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfec3aee-387b-4b53-8e48-760e99709d2e/config-reloader/0.log"
Jul 28 03:09:12.629359 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.629327    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfec3aee-387b-4b53-8e48-760e99709d2e/thanos-sidecar/0.log"
Jul 28 03:09:12.653743 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.653713    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfec3aee-387b-4b53-8e48-760e99709d2e/kube-rbac-proxy-web/0.log"
Jul 28 03:09:12.677478 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.677438    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfec3aee-387b-4b53-8e48-760e99709d2e/kube-rbac-proxy/0.log"
Jul 28 03:09:12.702158 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.702125    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfec3aee-387b-4b53-8e48-760e99709d2e/kube-rbac-proxy-thanos/0.log"
Jul 28 03:09:12.724338 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.724296    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bfec3aee-387b-4b53-8e48-760e99709d2e/init-config-reloader/0.log"
Jul 28 03:09:12.761680 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.761645    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-klb4j_196cacb9-9694-4493-a8e6-e41db2ce721f/prometheus-operator/0.log"
Jul 28 03:09:12.781889 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.781813    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-klb4j_196cacb9-9694-4493-a8e6-e41db2ce721f/kube-rbac-proxy/0.log"
Jul 28 03:09:12.809926 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.809894    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-wxf69_b3303926-a6d6-4c36-9a9d-4f048950bc56/prometheus-operator-admission-webhook/0.log"
Jul 28 03:09:12.931324 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.931249    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5988bb7987-nsstg_e1ce6711-b7fb-4482-a04d-3b4747e45c99/thanos-query/0.log"
Jul 28 03:09:12.954265 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.954229    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5988bb7987-nsstg_e1ce6711-b7fb-4482-a04d-3b4747e45c99/kube-rbac-proxy-web/0.log"
Jul 28 03:09:12.979037 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:12.979007    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5988bb7987-nsstg_e1ce6711-b7fb-4482-a04d-3b4747e45c99/kube-rbac-proxy/0.log"
Jul 28 03:09:13.001206 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:13.001177    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5988bb7987-nsstg_e1ce6711-b7fb-4482-a04d-3b4747e45c99/prom-label-proxy/0.log"
Jul 28 03:09:13.023463 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:13.023430    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5988bb7987-nsstg_e1ce6711-b7fb-4482-a04d-3b4747e45c99/kube-rbac-proxy-rules/0.log"
Jul 28 03:09:13.047802 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:13.047775    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5988bb7987-nsstg_e1ce6711-b7fb-4482-a04d-3b4747e45c99/kube-rbac-proxy-metrics/0.log"
Jul 28 03:09:14.276721 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:14.276685    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-q5n5g_dc1ce8f6-c11b-4e90-89b3-356006dc583d/networking-console-plugin/0.log"
Jul 28 03:09:15.058778 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.058743    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"]
Jul 28 03:09:15.059532 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.059507    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" containerName="copy"
Jul 28 03:09:15.059720 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.059704    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" containerName="copy"
Jul 28 03:09:15.059899 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.059883    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" containerName="gather"
Jul 28 03:09:15.059990 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.059980    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" containerName="gather"
Jul 28 03:09:15.060172 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.060152    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" containerName="copy"
Jul 28 03:09:15.060258 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.060176    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd76eea0-7751-4d83-982a-acddc0556d53" containerName="gather"
Jul 28 03:09:15.063711 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.063691    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.066113 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.066091    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6q4dh\"/\"default-dockercfg-fvqfh\""
Jul 28 03:09:15.070575 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.070527    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"]
Jul 28 03:09:15.182506 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.182469    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-proc\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.182833 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.182807    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-lib-modules\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.182998 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.182979    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-podres\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.183114 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.183100    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2g4r\" (UniqueName: \"kubernetes.io/projected/f2afe2fc-1cad-4bb1-900a-7e448b30c924-kube-api-access-w2g4r\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.183271 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.183235    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-sys\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.277934 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.277907    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-866cdddcb6-mbxzp_00ee1a01-9842-47cf-ab97-fefd4e1ef0ca/console/0.log"
Jul 28 03:09:15.284151 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.284117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-lib-modules\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.284303 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.284155    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-podres\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.284303 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.284175    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2g4r\" (UniqueName: \"kubernetes.io/projected/f2afe2fc-1cad-4bb1-900a-7e448b30c924-kube-api-access-w2g4r\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.284303 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.284240    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-sys\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.284303 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.284297    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-podres\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.284503 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.284301    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-lib-modules\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.284503 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.284349    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-sys\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.284503 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.284362    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-proc\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.284503 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.284397    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2afe2fc-1cad-4bb1-900a-7e448b30c924-proc\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.292089 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.292067    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2g4r\" (UniqueName: \"kubernetes.io/projected/f2afe2fc-1cad-4bb1-900a-7e448b30c924-kube-api-access-w2g4r\") pod \"perf-node-gather-daemonset-mvrdr\" (UID: \"f2afe2fc-1cad-4bb1-900a-7e448b30c924\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.321806 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.321777    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-98g8r_2789e7e3-91a1-447d-adc1-75c92a79ffa9/download-server/0.log"
Jul 28 03:09:15.377026 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.376983    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:15.532549 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.532509    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"]
Jul 28 03:09:15.539232 ip-10-0-140-37 kubenswrapper[2556]: W0728 03:09:15.539202    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf2afe2fc_1cad_4bb1_900a_7e448b30c924.slice/crio-032d8130e8fb5d36a0d0edeaad70db820cd0cb16e8c320719bbd8e8887a6fffe WatchSource:0}: Error finding container 032d8130e8fb5d36a0d0edeaad70db820cd0cb16e8c320719bbd8e8887a6fffe: Status 404 returned error can't find the container with id 032d8130e8fb5d36a0d0edeaad70db820cd0cb16e8c320719bbd8e8887a6fffe
Jul 28 03:09:15.901341 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:15.901247    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-xblm8_bb2c5c28-a06f-4359-9aa2-5681f3731f8d/volume-data-source-validator/0.log"
Jul 28 03:09:16.098790 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:16.098749    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr" event={"ID":"f2afe2fc-1cad-4bb1-900a-7e448b30c924","Type":"ContainerStarted","Data":"ce66bd7c853362b6bb3d84f5a5a1558f4c629e7c2d7edfee322c1e4b7d686164"}
Jul 28 03:09:16.098975 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:16.098823    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr" event={"ID":"f2afe2fc-1cad-4bb1-900a-7e448b30c924","Type":"ContainerStarted","Data":"032d8130e8fb5d36a0d0edeaad70db820cd0cb16e8c320719bbd8e8887a6fffe"}
Jul 28 03:09:16.098975 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:16.098869    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:16.115551 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:16.115489    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr" podStartSLOduration=1.115467733 podStartE2EDuration="1.115467733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 03:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:09:16.114055871 +0000 UTC m=+6236.286801121" watchObservedRunningTime="2026-07-28 03:09:16.115467733 +0000 UTC m=+6236.288212983"
Jul 28 03:09:16.879249 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:16.879217    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-79nrr_b87279e4-7169-4b8a-a1d7-bf1ff52a08ea/dns/0.log"
Jul 28 03:09:16.903124 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:16.903096    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-79nrr_b87279e4-7169-4b8a-a1d7-bf1ff52a08ea/kube-rbac-proxy/0.log"
Jul 28 03:09:17.111196 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:17.111165    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pjclb_cc5ba0f8-327b-4257-97da-632040885d84/dns-node-resolver/0.log"
Jul 28 03:09:17.792981 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:17.792949    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-st5g5_56abc673-f9ba-4b33-a82f-d0469ba37eb5/node-ca/0.log"
Jul 28 03:09:19.552999 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:19.552973    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x4n5c_a30566ed-1cc6-4db8-87f1-252244af5aef/serve-healthcheck-canary/0.log"
Jul 28 03:09:20.034352 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:20.034317    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dzw8g_7553c92a-8e6b-48a4-9ccc-a26280163ff1/kube-rbac-proxy/0.log"
Jul 28 03:09:20.060276 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:20.060227    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dzw8g_7553c92a-8e6b-48a4-9ccc-a26280163ff1/exporter/0.log"
Jul 28 03:09:20.081803 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:20.081760    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dzw8g_7553c92a-8e6b-48a4-9ccc-a26280163ff1/extractor/0.log"
Jul 28 03:09:22.118974 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:22.118943    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-mvrdr"
Jul 28 03:09:23.092993 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:23.092961    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78db6c7dfd-p54hf_776833e5-d7ec-464b-aa02-611e28dc1c10/manager/0.log"
Jul 28 03:09:23.918794 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:23.918759    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6fcdf65d46-s9w4q_f797e525-5eb0-46fb-8823-eb578ad552bf/manager/0.log"
Jul 28 03:09:31.164602 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:31.164573    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4cdnz_2305f190-18a0-46c6-a291-264be71b6d0d/kube-multus-additional-cni-plugins/0.log"
Jul 28 03:09:31.186315 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:31.186285    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4cdnz_2305f190-18a0-46c6-a291-264be71b6d0d/egress-router-binary-copy/0.log"
Jul 28 03:09:31.207366 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:31.207339    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4cdnz_2305f190-18a0-46c6-a291-264be71b6d0d/cni-plugins/0.log"
Jul 28 03:09:31.228412 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:31.228389    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4cdnz_2305f190-18a0-46c6-a291-264be71b6d0d/bond-cni-plugin/0.log"
Jul 28 03:09:31.249436 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:31.249412    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4cdnz_2305f190-18a0-46c6-a291-264be71b6d0d/routeoverride-cni/0.log"
Jul 28 03:09:31.270797 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:31.270775    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4cdnz_2305f190-18a0-46c6-a291-264be71b6d0d/whereabouts-cni-bincopy/0.log"
Jul 28 03:09:31.302754 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:31.302727    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4cdnz_2305f190-18a0-46c6-a291-264be71b6d0d/whereabouts-cni/0.log"
Jul 28 03:09:31.946546 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:31.946519    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hgt5v_8c6d42fc-71f1-466d-9183-521c2b330a67/kube-multus/0.log"
Jul 28 03:09:32.022660 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:32.022629    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9qd9t_294ecef6-7cbe-446d-96f6-36de1ab923aa/network-metrics-daemon/0.log"
Jul 28 03:09:32.055726 ip-10-0-140-37 kubenswrapper[2556]: I0728 03:09:32.055695    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9qd9t_294ecef6-7cbe-446d-96f6-36de1ab923aa/kube-rbac-proxy/0.log"