Jul 29 10:28:45.510697 ip-10-0-129-80 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 10:28:45.824196 ip-10-0-129-80 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:45.824196 ip-10-0-129-80 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 10:28:45.824196 ip-10-0-129-80 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:45.824196 ip-10-0-129-80 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 10:28:45.824196 ip-10-0-129-80 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:45.827377 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.827286    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 10:28:45.835880 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835862    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:45.835880 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835879    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835884    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835887    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835890    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835893    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835896    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835898    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835901    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835904    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835906    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835915    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835918    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835920    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835923    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835925    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835929    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835931    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835933    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835936    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835939    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:45.835959 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835941    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835944    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835946    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835949    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835951    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835954    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835957    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835960    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835963    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835966    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835968    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835971    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835974    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835976    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835979    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835982    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835985    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835987    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835990    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835992    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:45.836447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835995    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.835997    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836000    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836003    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836005    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836008    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836011    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836014    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836016    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836019    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836022    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836024    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836026    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836029    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836034    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836038    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836041    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836043    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836046    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:45.836933 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836048    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836051    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836053    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836056    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836059    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836061    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836063    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836066    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836068    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836071    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836073    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836076    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836078    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836081    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836084    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836087    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836089    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836092    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836096    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836101    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:45.837401 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836105    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836108    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836111    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836114    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836116    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836119    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836123    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836127    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836130    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836133    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836136    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836139    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836142    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836145    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836147    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836150    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836153    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836155    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836158    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:45.838091 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836914    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836921    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836924    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836927    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836930    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836933    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836936    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836939    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836942    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836945    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836949    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836952    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836955    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836958    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836961    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836965    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836967    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836970    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836972    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836976    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:45.838844 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836981    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836986    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836989    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836992    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836994    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.836997    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837000    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837003    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837006    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837009    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837012    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837014    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837017    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837019    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837022    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837025    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837027    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837031    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837035    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:45.839367 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837038    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837041    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837044    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837046    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837049    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837052    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837056    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837059    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837062    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837065    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837067    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837070    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837074    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837077    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837080    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837082    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837085    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837088    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837090    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837092    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:45.839859 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837095    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837097    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837100    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837102    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837105    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837107    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837110    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837112    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837115    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837117    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837119    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837122    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837124    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837127    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837129    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837132    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837134    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837137    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837140    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837144    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:45.840343 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837146    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837149    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837151    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837154    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837156    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837159    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837161    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837164    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837166    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837169    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837171    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837174    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837176    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837179    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837181    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837183    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837186    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837188    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837191    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.837193    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:45.840952 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838439    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838454    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838459    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838463    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838468    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838479    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838486    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838490    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838495    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838500    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838504    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838508    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838512    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838517    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838521    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838526    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838530    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838538    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838542    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:45.841447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838547    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838552    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838555    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838610    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838617    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838623    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838628    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838633    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838638    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838644    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838648    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838653    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838658    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838663    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838668    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838700    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838705    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838708    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838710    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838713    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838716    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:45.841917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838719    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838722    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838730    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838733    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838875    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838879    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838882    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838885    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838888    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838891    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838893    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838896    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838899    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838901    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838905    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838909    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838911    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838917    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838920    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:45.842434 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838923    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838925    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838928    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838931    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838933    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838936    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838939    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838941    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838944    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838947    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838949    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838951    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838954    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838957    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838960    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838963    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838965    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838968    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838972    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:45.842910 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838975    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838978    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838981    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838984    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838986    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838991    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838993    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838996    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.838999    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839002    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839005    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839007    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839010    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839013    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839017    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839021    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839024    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839027    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839030    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:45.843385 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839033    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839035    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839108    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839116    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839122    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839127    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839132    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839135    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839139    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839143    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839146    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839150    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839159    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839162    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839165    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839168    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839171    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839174    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839176    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839179    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839185    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839188    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839192    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839196    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 10:28:45.843905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839199    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839203    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839206    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839209    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839212    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839215    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839218    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839221    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839224    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839227    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839231    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839234    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839237    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839240    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839242    2569 flags.go:64] FLAG: --enable-server="true"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839245    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839250    2569 flags.go:64] FLAG: --event-burst="100"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839253    2569 flags.go:64] FLAG: --event-qps="50"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839256    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839259    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839264    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839268    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839271    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839274    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839276    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 29 10:28:45.844503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839279    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839282    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839285    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839297    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839301    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839304    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839306    2569 flags.go:64] FLAG: --feature-gates=""
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839310    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839313    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839316    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839319    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839322    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839325    2569 flags.go:64] FLAG: --help="false"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839328    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-129-80.ec2.internal"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839331    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839334    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839337    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839340    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839343    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839346    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839349    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839352    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839354    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 10:28:45.845108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839357    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839360    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839363    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839366    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839370    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839373    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839376    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839379    2569 flags.go:64] FLAG: --lock-file=""
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839382    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839385    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839388    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839393    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839396    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839400    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839403    2569 flags.go:64] FLAG: --logging-format="text"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839406    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839409    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839412    2569 flags.go:64] FLAG: --manifest-url=""
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839415    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839419    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839422    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839443    2569 flags.go:64] FLAG: --max-pods="110"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839447    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839450    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839452    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 10:28:45.845723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839456    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839459    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839462    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839464    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839473    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839476    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839479    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839482    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839485    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839490    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839493    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839497    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839500    2569 flags.go:64] FLAG: --port="10250"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839503    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839507    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-074a4cd53c23e63f6"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839511    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839514    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839517    2569 flags.go:64] FLAG: --register-node="true"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839520    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839523    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839526    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839530    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839533    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839536    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839539    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 10:28:45.846334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839542    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839545    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839548    2569 flags.go:64] FLAG: --runonce="false"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839551    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839554    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839557    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839559    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839562    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839565    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839568    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839571    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839574    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839577    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839580    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839582    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839585    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839588    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839591    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839598    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839600    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839603    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839609    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839611    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839614    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839617    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 10:28:45.846970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839620    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839622    2569 flags.go:64] FLAG: --v="2"
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839626    2569 flags.go:64] FLAG: --version="false"
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839630    2569 flags.go:64] FLAG: --vmodule=""
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839634    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.839637    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839746    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839750    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839753    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839756    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839759    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839761    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839764    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839767    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839770    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839774    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839777    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839780    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839782    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839785    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839788    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:45.847659 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839790    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839792    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839795    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839798    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839802    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839804    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839807    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839811    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839813    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839816    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839819    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839822    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839824    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839827    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839829    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839832    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839835    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839838    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839840    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839843    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:45.848165 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839846    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839848    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839851    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839854    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839856    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839859    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839861    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839864    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839867    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839869    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839873    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839877    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839880    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839883    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839886    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839889    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839893    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839895    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839898    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:45.848670 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839903    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839905    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839908    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839911    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839913    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839916    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839919    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839921    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839924    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839926    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839929    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839932    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839934    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839937    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839939    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839942    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839944    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839947    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839950    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839953    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:45.849134 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839955    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839958    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839960    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839963    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839965    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839968    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839970    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839973    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839976    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839978    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839981    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839983    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839989    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839993    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839996    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.839999    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.840002    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.840004    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.840007    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.840010    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:45.849639 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.840012    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.840015    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.840017    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.840020    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.840023    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.840599    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.847158    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.847172    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.847191    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.847200    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847254    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847259    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847263    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847266    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847269    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:45.850118 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847271    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847274    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847277    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847280    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847282    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847285    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847288    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847300    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847304    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847306    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847309    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847312    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847315    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847318    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847320    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847324    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847328    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847331    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847334    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847337    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:45.850541 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847344    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847347    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847350    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847353    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847356    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847360    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847364    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847367    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847370    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847373    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847376    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847379    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847382    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847384    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847387    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847390    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847393    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847396    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847399    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847402    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:45.851021 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847405    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847408    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847411    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847413    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847416    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847419    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847421    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847436    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847439    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847442    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847445    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847448    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847451    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847454    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847457    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847460    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847463    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847466    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847469    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847471    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:45.851532 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847475    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847477    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847480    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847483    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847485    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847488    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847490    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847493    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847496    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847498    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847501    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847503    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847506    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847508    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847511    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847513    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847516    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847519    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847521    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847524    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:45.852016 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847526    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847528    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847531    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847533    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847536    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847539    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847542    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847545    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847548    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847551    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847554    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847557    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847559    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847563    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.847569    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:45.852512 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847680    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847684    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847687    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847690    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847693    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847695    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847699    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847701    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847704    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847707    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847710    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847713    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847715    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847718    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847721    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847724    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847727    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847729    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847732    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847734    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:45.852919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847737    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847739    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847742    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847745    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847748    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847752    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847754    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847757    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847760    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847762    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847765    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847767    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847771    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847774    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847777    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847779    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847782    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847784    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847787    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847790    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:45.853409 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847792    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847795    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847798    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847800    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847803    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847807    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847810    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847813    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847816    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847819    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847822    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847825    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847827    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847830    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847832    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847836    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847838    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847841    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847844    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:45.853918 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847847    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847851    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847854    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847857    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847860    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847863    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847866    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847868    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847871    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847874    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847876    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847879    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847882    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847884    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847887    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847889    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847892    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847894    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847897    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847899    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:45.854396 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847902    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847904    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847907    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847909    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847912    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847914    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847917    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847919    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847922    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847924    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847927    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847930    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847933    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847935    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847938    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847940    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847943    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847945    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847948    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:45.854898 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:45.847951    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:45.855367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.847956    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:45.855367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.848077    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 10:28:45.855367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.850819    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 10:28:45.855367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.851949    2569 server.go:1019] "Starting client certificate rotation"
Jul 29 10:28:45.855367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.852054    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:45.855367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.852091    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:45.855367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.854224    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:45.855623 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.855579    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:45.864658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.864643    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 29 10:28:45.866555 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.866541    2569 log.go:25] "Validated CRI v1 image API"
Jul 29 10:28:45.871630 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.871273    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 10:28:45.875133 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.875110    2569 fs.go:135] Filesystem UUIDs: map[11305017-5f3e-4e40-be76-7ddd237878d6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8f88e750-c5c4-4c3a-9867-2a88c5f626c4:/dev/nvme0n1p3]
Jul 29 10:28:45.875213 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.875131    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 10:28:45.879858 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.879751    2569 manager.go:216] Machine: {Timestamp:2026-07-29 10:28:45.878782382 +0000 UTC m=+0.271995243 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097793 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28020a7ed8b9ecd4d7619892dffdcb SystemUUID:ec28020a-7ed8-b9ec-d4d7-619892dffdcb BootID:39421415-dce1-46a4-ab8c-8bbe2af34e89 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ca:99:6c:6f:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ca:99:6c:6f:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:33:be:d4:fd:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 10:28:45.879858 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.879853    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 10:28:45.879964 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.879927    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 10:28:45.880777 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.880753    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 10:28:45.880905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.880780    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 10:28:45.881814 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.881802    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 10:28:45.881848 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.881817    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 10:28:45.881848 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.881825    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:45.882402 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.882390    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:45.883569 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.883558    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 10:28:45.883995 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.883977    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:45.886311 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.886300    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:45.886419 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.886410    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 10:28:45.886467 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.886454    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 10:28:45.886467 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.886464    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 10:28:45.886528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.886480    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 10:28:45.886528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.886489    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 29 10:28:45.886528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.886499    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 10:28:45.886846 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.886821    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:45.886846 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.886839    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:45.888677 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.888655    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 10:28:45.889978 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.889964    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 10:28:45.890027 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.889995    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 10:28:45.890678 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890667    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 10:28:45.890718 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890684    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 10:28:45.890718 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890690    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 10:28:45.890718 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890696    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 10:28:45.890718 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890705    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 10:28:45.890718 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890711    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 10:28:45.890718 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890720    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 10:28:45.890872 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890725    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 10:28:45.890872 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890731    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 10:28:45.890872 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890737    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 10:28:45.890872 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890745    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 10:28:45.890872 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.890751    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 10:28:45.891340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.891330    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 10:28:45.891340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.891340    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 10:28:45.894443 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.894416    2569 server.go:1268] "Started kubelet"
Jul 29 10:28:45.894676 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.894620    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 10:28:45.894771 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.894696    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 10:28:45.894771 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.894715    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 10:28:45.895310 ip-10-0-129-80 systemd[1]: Started Kubernetes Kubelet.
Jul 29 10:28:45.895708 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.895568    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 10:28:45.896140 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.896108    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 10:28:45.896140 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.896115    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 10:28:45.896706 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.896687    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 10:28:45.896808 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.896795    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 10:28:45.899582 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.898690    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-80.ec2.internal.18c6bbec4e01a109  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-80.ec2.internal,UID:ip-10-0-129-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-80.ec2.internal,},FirstTimestamp:2026-07-29 10:28:45.894385929 +0000 UTC m=+0.287598790,LastTimestamp:2026-07-29 10:28:45.894385929 +0000 UTC m=+0.287598790,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-80.ec2.internal,}"
Jul 29 10:28:45.900816 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.900800    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 10:28:45.902347 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.902333    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5p7gt"
Jul 29 10:28:45.902787 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.902769    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 10:28:45.902787 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.902776    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:45.903145 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.902882    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:45.903596 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.903480    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 10:28:45.903679 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.903489    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 10:28:45.903679 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.903649    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 10:28:45.903810 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.903793    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 10:28:45.903810 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.903807    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 10:28:45.904935 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.904698    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 10:28:45.904935 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.904893    2569 factory.go:55] Registering systemd factory
Jul 29 10:28:45.904935 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.904904    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 10:28:45.904935 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.904922    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 29 10:28:45.905451 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.905421    2569 factory.go:153] Registering CRI-O factory
Jul 29 10:28:45.905542 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.905533    2569 factory.go:223] Registration of the crio container factory successfully
Jul 29 10:28:45.905690 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.905680    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 10:28:45.905798 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.905774    2569 factory.go:103] Registering Raw factory
Jul 29 10:28:45.905877 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.905870    2569 manager.go:1196] Started watching for new ooms in manager
Jul 29 10:28:45.906465 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.906449    2569 manager.go:349] Starting recovery of all containers
Jul 29 10:28:45.910175 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.910152    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5p7gt"
Jul 29 10:28:45.916668 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.916652    2569 manager.go:354] Recovery completed
Jul 29 10:28:45.921250 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.921238    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:45.923149 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.923132    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:45.923245 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.923165    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:45.923245 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.923181    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:45.923875 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.923862    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 10:28:45.923875 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.923876    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 10:28:45.924006 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.923893    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:45.924006 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.923865    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:45.924662 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.924649    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:45.924708 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.924673    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:45.924708 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.924683    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:45.924941 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.924929    2569 policy_none.go:49] "None policy: Start"
Jul 29 10:28:45.924996 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.924948    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 10:28:45.924996 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.924962    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 10:28:45.926095 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.926082    2569 policy_none.go:47] "Start"
Jul 29 10:28:45.972056 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.971864    2569 manager.go:337] "Starting Device Plugin manager"
Jul 29 10:28:45.972146 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.972085    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 10:28:45.972146 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.972096    2569 server.go:87] "Starting device plugin registration server"
Jul 29 10:28:45.972330 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.972319    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 10:28:45.972365 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.972334    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 10:28:45.972464 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.972449    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 10:28:45.972560 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.972548    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 10:28:45.972623 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:45.972561    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 10:28:45.973595 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.973578    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 10:28:45.973651 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:45.973622    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.034065 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.034022    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 10:28:46.036153 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.035175    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 10:28:46.036153 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.035211    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 10:28:46.036153 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.035235    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 10:28:46.036153 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.035274    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 10:28:46.037186 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.037166    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:46.073005 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.072969    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:46.074131 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.074114    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:46.074245 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.074147    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:46.074245 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.074160    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:46.074245 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.074186    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.079745 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.079731    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.079786 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.079754    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-80.ec2.internal\": node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.096684 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.096657    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.135800 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.135771    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal"]
Jul 29 10:28:46.135868 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.135841    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:46.137335 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.137319    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:46.137399 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.137350    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:46.137399 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.137362    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:46.139607 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.139594    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:46.139744 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.139730    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.139777 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.139762    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:46.140309 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.140295    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:46.140359 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.140326    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:46.140359 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.140338    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:46.140452 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.140295    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:46.140452 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.140389    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:46.140452 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.140403    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:46.142550 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.142537    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.142593 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.142562    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:46.143214 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.143194    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:46.143321 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.143225    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:46.143321 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.143239    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:46.159095 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.159066    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-80.ec2.internal\" not found" node="ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.163270 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.163252    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-80.ec2.internal\" not found" node="ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.197005 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.196978    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.205149 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.205126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.205206 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.205154    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.205206 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.205173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f186ee96a5b10264dea34206a4faa43e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-80.ec2.internal\" (UID: \"f186ee96a5b10264dea34206a4faa43e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.297749 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.297715    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.306068 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.306045    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.306177 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.306074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f186ee96a5b10264dea34206a4faa43e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-80.ec2.internal\" (UID: \"f186ee96a5b10264dea34206a4faa43e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.306177 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.306090    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.306177 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.306142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f186ee96a5b10264dea34206a4faa43e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-80.ec2.internal\" (UID: \"f186ee96a5b10264dea34206a4faa43e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.306177 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.306161    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.306314 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.306217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483318e6ca30ead09c5152f256a0232-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"0483318e6ca30ead09c5152f256a0232\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.398525 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.398491    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.462871 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.462837    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.466949 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.466925    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:46.499106 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.499069    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.602354 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.602312    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.704160 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.704056    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.804582 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.804549    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.853404 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.853366    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 10:28:46.853834 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.853573    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 10:28:46.904978 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:46.904944    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:46.914391 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.914356    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 10:23:46 +0000 UTC" deadline="2028-02-08 20:59:10.768514746 +0000 UTC"
Jul 29 10:28:46.914391 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:46.914387    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13426h30m23.854132064s"
Jul 29 10:28:47.005685 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:47.005607    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:47.105905 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:47.105859    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:47.127728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.127702    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:47.179926 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:47.179887    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0483318e6ca30ead09c5152f256a0232.slice/crio-84e239b046b671557d756a59a5d3c54e5016f1c984cc58a8731ab89c9b119b59 WatchSource:0}: Error finding container 84e239b046b671557d756a59a5d3c54e5016f1c984cc58a8731ab89c9b119b59: Status 404 returned error can't find the container with id 84e239b046b671557d756a59a5d3c54e5016f1c984cc58a8731ab89c9b119b59
Jul 29 10:28:47.184260 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.184240    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:28:47.206715 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:47.206693    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:47.309087 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:47.308999    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:47.344820 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.344794    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:47.409474 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:47.409413    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found"
Jul 29 10:28:47.483856 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.483822    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:47.498812 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:47.498784    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf186ee96a5b10264dea34206a4faa43e.slice/crio-8da59f8285bb8c056ffb3db1e344ed82dbadbec8403212d6b1aa8352587988c8 WatchSource:0}: Error finding container 8da59f8285bb8c056ffb3db1e344ed82dbadbec8403212d6b1aa8352587988c8: Status 404 returned error can't find the container with id 8da59f8285bb8c056ffb3db1e344ed82dbadbec8403212d6b1aa8352587988c8
Jul 29 10:28:47.503306 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.503285    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:47.510957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.510936    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:47.511794 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.511780    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal"
Jul 29 10:28:47.522495 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.522476    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:47.888254 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.888223    2569 apiserver.go:52] "Watching apiserver"
Jul 29 10:28:47.896547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.896523    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 10:28:47.896986 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.896958    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal","openshift-multus/multus-6nndf","openshift-multus/multus-additional-cni-plugins-45d2k","openshift-network-operator/iptables-alerter-dvqr7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns","openshift-cluster-node-tuning-operator/tuned-rrkxz","openshift-image-registry/node-ca-8gj9q","openshift-multus/network-metrics-daemon-ddfml","openshift-network-diagnostics/network-check-target-kbgr9","openshift-ovn-kubernetes/ovnkube-node-tpvc8","kube-system/global-pull-secret-syncer-r778m","kube-system/konnectivity-agent-2kzz5","kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal","openshift-dns/node-resolver-57dwz"]
Jul 29 10:28:47.899907 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.899886    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:47.902232 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.902207    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:47.902369 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:47.902287    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:28:47.903227 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.903207    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:47.904004 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.903751    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 10:28:47.904004 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.903775    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j6p8b\""
Jul 29 10:28:47.904004 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.903757    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 10:28:47.904004 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.903755    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 10:28:47.905090 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.904903    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:47.907524 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.907505    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:47.907712 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.907691    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 10:28:47.907889 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.907868    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 10:28:47.908287 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.908254    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dqcsf\""
Jul 29 10:28:47.908913 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.908271    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 10:28:47.909186 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.909165    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 10:28:47.909548 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.909214    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 10:28:47.910242 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.910220    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:47.911110 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.911086    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 10:28:47.911326 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.911303    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nqlh9\""
Jul 29 10:28:47.911413 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.911396    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:47.911651 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.911631    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:47.913929 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.913897    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:47.914884 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.914863    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 10:28:47.915140 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915119    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 10:28:47.915255 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915230    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8z2\" (UniqueName: \"kubernetes.io/projected/a9a64c38-a6c8-4d27-9d16-77605f9f5742-kube-api-access-vv8z2\") pod \"node-ca-8gj9q\" (UID: \"a9a64c38-a6c8-4d27-9d16-77605f9f5742\") " pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:47.915334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32248a0a-b444-475d-8a21-2139f1dde3fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:47.915334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915302    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9a64c38-a6c8-4d27-9d16-77605f9f5742-serviceca\") pod \"node-ca-8gj9q\" (UID: \"a9a64c38-a6c8-4d27-9d16-77605f9f5742\") " pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:47.915334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915325    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2285452e-2515-4e83-8aa2-6334577e0140-dbus\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:47.915547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915343    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-system-cni-dir\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:47.915547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:47.915547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32248a0a-b444-475d-8a21-2139f1dde3fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:47.915547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915427    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9a64c38-a6c8-4d27-9d16-77605f9f5742-host\") pod \"node-ca-8gj9q\" (UID: \"a9a64c38-a6c8-4d27-9d16-77605f9f5742\") " pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:47.915547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915479    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 10:28:47.915547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2285452e-2515-4e83-8aa2-6334577e0140-kubelet-config\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:47.915547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915517    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-cnibin\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:47.915547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915543    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32248a0a-b444-475d-8a21-2139f1dde3fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:47.915890 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915556    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mjgpl\""
Jul 29 10:28:47.915890 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915567    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzvh\" (UniqueName: \"kubernetes.io/projected/32248a0a-b444-475d-8a21-2139f1dde3fa-kube-api-access-zbzvh\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:47.915890 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-os-release\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:47.915890 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.915619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:47.916446 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.916415    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:47.916625 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.916607    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:47.916725 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:47.916695    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:28:47.917127 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.917108    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:47.917222 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.917201    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rkjxd\""
Jul 29 10:28:47.917933 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.917911    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:47.919436 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.919394    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:47.921977 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.921953    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6nndf"
Jul 29 10:28:47.922394 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.922368    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 10:28:47.922492 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.922400    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7xnhw\""
Jul 29 10:28:47.922492 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.922430    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 10:28:47.924336 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.924279    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 10:28:47.924529 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.924511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:47.924890 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.924685    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mvcrt\""
Jul 29 10:28:47.927132 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.927025    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 10:28:47.927945 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.927402    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 10:28:47.927945 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.927519    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 10:28:47.927945 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.927650    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 10:28:47.927945 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.927726    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 10:28:47.927945 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.927837    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5p886\""
Jul 29 10:28:47.928471 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.928449    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:47.928570 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:47.928539    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:28:47.928971 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.928950    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 10:28:47.930744 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.930705    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:28:47.933399 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.933379    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z4xxv\""
Jul 29 10:28:47.933566 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.933459    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 10:28:47.933707 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.933692    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 10:28:47.950033 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.950008    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xvlx8"
Jul 29 10:28:47.960852 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:47.960487    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xvlx8"
Jul 29 10:28:48.005066 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.005037    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 10:28:48.016680 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.016624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-cni-dir\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.016866 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.016806    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d83cf8-33cd-4193-b820-b0b81302862d-host-slash\") pod \"iptables-alerter-dvqr7\" (UID: \"05d83cf8-33cd-4193-b820-b0b81302862d\") " pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.016926 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.016861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-kubelet\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.016926 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.016897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-ovnkube-script-lib\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.017024 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.016952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32248a0a-b444-475d-8a21-2139f1dde3fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.017024 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.016995    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9a64c38-a6c8-4d27-9d16-77605f9f5742-host\") pod \"node-ca-8gj9q\" (UID: \"a9a64c38-a6c8-4d27-9d16-77605f9f5742\") " pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:48.017126 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017025    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2285452e-2515-4e83-8aa2-6334577e0140-kubelet-config\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:48.017126 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017059    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c50ec00e-12f4-41e9-be76-5ab3c1571fad-agent-certs\") pod \"konnectivity-agent-2kzz5\" (UID: \"c50ec00e-12f4-41e9-be76-5ab3c1571fad\") " pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:28:48.017126 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017095    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-systemd-units\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.017271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017132    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32248a0a-b444-475d-8a21-2139f1dde3fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.017271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017169    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzvh\" (UniqueName: \"kubernetes.io/projected/32248a0a-b444-475d-8a21-2139f1dde3fa-kube-api-access-zbzvh\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.017271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017210    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-sysctl-d\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.017271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-cnibin\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.017463 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-conf-dir\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.017463 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:48.017463 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017341    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-run-ovn-kubernetes\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.017463 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017384    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:48.017463 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017433    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-run-multus-certs\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.017705 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017492    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-cni-netd\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.017536    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018112    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-ovnkube-config\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018156    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-host\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32248a0a-b444-475d-8a21-2139f1dde3fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9a64c38-a6c8-4d27-9d16-77605f9f5742-serviceca\") pod \"node-ca-8gj9q\" (UID: \"a9a64c38-a6c8-4d27-9d16-77605f9f5742\") " pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-etc-selinux\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018299    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-var-lib-kubelet\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018334    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqs5b\" (UniqueName: \"kubernetes.io/projected/05d83cf8-33cd-4193-b820-b0b81302862d-kube-api-access-rqs5b\") pod \"iptables-alerter-dvqr7\" (UID: \"05d83cf8-33cd-4193-b820-b0b81302862d\") " pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-system-cni-dir\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018401    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpn5t\" (UniqueName: \"kubernetes.io/projected/d14e709b-eb10-4251-80dc-54e1aa91e1dd-kube-api-access-rpn5t\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018438    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-run-netns\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.018450    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-cnibin\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018518    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c50ec00e-12f4-41e9-be76-5ab3c1571fad-konnectivity-ca\") pod \"konnectivity-agent-2kzz5\" (UID: \"c50ec00e-12f4-41e9-be76-5ab3c1571fad\") " pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018543    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32248a0a-b444-475d-8a21-2139f1dde3fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018554    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-sysctl-conf\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.019741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018588    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-cni-bin\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-ovn-node-metrics-cert\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018654    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9a64c38-a6c8-4d27-9d16-77605f9f5742-host\") pod \"node-ca-8gj9q\" (UID: \"a9a64c38-a6c8-4d27-9d16-77605f9f5742\") " pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-os-release\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018722    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2285452e-2515-4e83-8aa2-6334577e0140-kubelet-config\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018765    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-os-release\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018796    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-system-cni-dir\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.018869    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret podName:2285452e-2515-4e83-8aa2-6334577e0140 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:48.518840833 +0000 UTC m=+2.912053688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret") pod "global-pull-secret-syncer-r778m" (UID: "2285452e-2515-4e83-8aa2-6334577e0140") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018893    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-socket-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.018924    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-cnibin\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019001    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-registration-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-os-release\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019078    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-var-lib-cni-bin\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019108    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-var-lib-cni-multus\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019137    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-etc-openvswitch\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019167    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-env-overrides\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.020768 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019217    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-sys-fs\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32248a0a-b444-475d-8a21-2139f1dde3fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-lib-modules\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019327    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzhx\" (UniqueName: \"kubernetes.io/projected/eb2a6946-777b-4f49-af81-0942eaffea47-kube-api-access-dvzhx\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019359    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-socket-dir-parent\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019404    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-var-lib-kubelet\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019433    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019470    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32248a0a-b444-475d-8a21-2139f1dde3fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-sysconfig\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-kubernetes\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019628    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb2a6946-777b-4f49-af81-0942eaffea47-tmp\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32248a0a-b444-475d-8a21-2139f1dde3fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-etc-kubernetes\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05d83cf8-33cd-4193-b820-b0b81302862d-iptables-alerter-script\") pod \"iptables-alerter-dvqr7\" (UID: \"05d83cf8-33cd-4193-b820-b0b81302862d\") " pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.021636 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019753    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9a64c38-a6c8-4d27-9d16-77605f9f5742-serviceca\") pod \"node-ca-8gj9q\" (UID: \"a9a64c38-a6c8-4d27-9d16-77605f9f5742\") " pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019800    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-var-lib-openvswitch\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-run-openvswitch\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019906    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-device-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-daemon-config\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.019986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjsrm\" (UniqueName: \"kubernetes.io/projected/262e51a4-76b2-4a18-9814-9dd42ccf84fd-kube-api-access-rjsrm\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020025    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhdb\" (UniqueName: \"kubernetes.io/projected/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-kube-api-access-7fhdb\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020060    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-log-socket\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020086    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/001cfcc8-dc86-4070-b7e7-f374fbae8581-hosts-file\") pod \"node-resolver-57dwz\" (UID: \"001cfcc8-dc86-4070-b7e7-f374fbae8581\") " pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001cfcc8-dc86-4070-b7e7-f374fbae8581-tmp-dir\") pod \"node-resolver-57dwz\" (UID: \"001cfcc8-dc86-4070-b7e7-f374fbae8581\") " pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020182    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhqp6\" (UniqueName: \"kubernetes.io/projected/c85e413e-151e-4792-8dfe-18d8b0b07ecf-kube-api-access-lhqp6\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020222    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8z2\" (UniqueName: \"kubernetes.io/projected/a9a64c38-a6c8-4d27-9d16-77605f9f5742-kube-api-access-vv8z2\") pod \"node-ca-8gj9q\" (UID: \"a9a64c38-a6c8-4d27-9d16-77605f9f5742\") " pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2285452e-2515-4e83-8aa2-6334577e0140-dbus\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020287    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-run-systemd\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-systemd\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb5dw\" (UniqueName: \"kubernetes.io/projected/001cfcc8-dc86-4070-b7e7-f374fbae8581-kube-api-access-rb5dw\") pod \"node-resolver-57dwz\" (UID: \"001cfcc8-dc86-4070-b7e7-f374fbae8581\") " pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-system-cni-dir\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.022457 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020424    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-run-ovn\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020541    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-modprobe-d\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020567    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-run-k8s-cni-cncf-io\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-run-netns\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-hostroot\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-slash\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-node-log\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2285452e-2515-4e83-8aa2-6334577e0140-dbus\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-sys\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.020970    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb2a6946-777b-4f49-af81-0942eaffea47-etc-tuned\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.021010    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/262e51a4-76b2-4a18-9814-9dd42ccf84fd-cni-binary-copy\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.023387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.021070    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-run\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.028901 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.028877    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 10:28:48.033637 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.033498    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8z2\" (UniqueName: \"kubernetes.io/projected/a9a64c38-a6c8-4d27-9d16-77605f9f5742-kube-api-access-vv8z2\") pod \"node-ca-8gj9q\" (UID: \"a9a64c38-a6c8-4d27-9d16-77605f9f5742\") " pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:48.033969 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.033943    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzvh\" (UniqueName: \"kubernetes.io/projected/32248a0a-b444-475d-8a21-2139f1dde3fa-kube-api-access-zbzvh\") pod \"multus-additional-cni-plugins-45d2k\" (UID: \"32248a0a-b444-475d-8a21-2139f1dde3fa\") " pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.040762 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.040697    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" event={"ID":"f186ee96a5b10264dea34206a4faa43e","Type":"ContainerStarted","Data":"8da59f8285bb8c056ffb3db1e344ed82dbadbec8403212d6b1aa8352587988c8"}
Jul 29 10:28:48.042270 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.042245    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" event={"ID":"0483318e6ca30ead09c5152f256a0232","Type":"ContainerStarted","Data":"84e239b046b671557d756a59a5d3c54e5016f1c984cc58a8731ab89c9b119b59"}
Jul 29 10:28:48.121741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-modprobe-d\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.121741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-run-k8s-cni-cncf-io\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121770    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-run-netns\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-hostroot\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-slash\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121846    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-node-log\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121864    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-modprobe-d\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-sys\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121903    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb2a6946-777b-4f49-af81-0942eaffea47-etc-tuned\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-run-netns\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121923    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-hostroot\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121958    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-node-log\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/262e51a4-76b2-4a18-9814-9dd42ccf84fd-cni-binary-copy\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121984    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-sys\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121999    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-slash\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-run\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-cni-dir\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122058    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d83cf8-33cd-4193-b820-b0b81302862d-host-slash\") pod \"iptables-alerter-dvqr7\" (UID: \"05d83cf8-33cd-4193-b820-b0b81302862d\") " pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-kubelet\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-ovnkube-script-lib\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c50ec00e-12f4-41e9-be76-5ab3c1571fad-agent-certs\") pod \"konnectivity-agent-2kzz5\" (UID: \"c50ec00e-12f4-41e9-be76-5ab3c1571fad\") " pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-systemd-units\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-sysctl-d\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-cnibin\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-conf-dir\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122266    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122292    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-run-ovn-kubernetes\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-run-multus-certs\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122377    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-cni-netd\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122406    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-ovnkube-config\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-host\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.122728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122524    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-etc-selinux\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122550    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-var-lib-kubelet\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122578    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqs5b\" (UniqueName: \"kubernetes.io/projected/05d83cf8-33cd-4193-b820-b0b81302862d-kube-api-access-rqs5b\") pod \"iptables-alerter-dvqr7\" (UID: \"05d83cf8-33cd-4193-b820-b0b81302862d\") " pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122607    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpn5t\" (UniqueName: \"kubernetes.io/projected/d14e709b-eb10-4251-80dc-54e1aa91e1dd-kube-api-access-rpn5t\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122615    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/262e51a4-76b2-4a18-9814-9dd42ccf84fd-cni-binary-copy\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122634    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-run-netns\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.121867    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-run-k8s-cni-cncf-io\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c50ec00e-12f4-41e9-be76-5ab3c1571fad-konnectivity-ca\") pod \"konnectivity-agent-2kzz5\" (UID: \"c50ec00e-12f4-41e9-be76-5ab3c1571fad\") " pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122705    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-run\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-host\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122757    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-sysctl-conf\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-cni-bin\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122818    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-ovn-node-metrics-cert\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-cni-dir\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-socket-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122879    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-registration-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-os-release\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.123638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-var-lib-cni-bin\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122915    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-kubelet\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-var-lib-cni-multus\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122981    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-var-lib-cni-bin\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123006    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-etc-openvswitch\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-env-overrides\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-sys-fs\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.123068    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123111    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-lib-modules\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzhx\" (UniqueName: \"kubernetes.io/projected/eb2a6946-777b-4f49-af81-0942eaffea47-kube-api-access-dvzhx\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123164    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-socket-dir-parent\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-var-lib-kubelet\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c50ec00e-12f4-41e9-be76-5ab3c1571fad-konnectivity-ca\") pod \"konnectivity-agent-2kzz5\" (UID: \"c50ec00e-12f4-41e9-be76-5ab3c1571fad\") " pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123223    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-sysconfig\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-kubernetes\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123309    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb2a6946-777b-4f49-af81-0942eaffea47-tmp\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.124476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123335    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-etc-kubernetes\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123362    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05d83cf8-33cd-4193-b820-b0b81302862d-iptables-alerter-script\") pod \"iptables-alerter-dvqr7\" (UID: \"05d83cf8-33cd-4193-b820-b0b81302862d\") " pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123391    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-var-lib-openvswitch\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-cni-netd\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-run-openvswitch\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-device-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-daemon-config\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123521    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjsrm\" (UniqueName: \"kubernetes.io/projected/262e51a4-76b2-4a18-9814-9dd42ccf84fd-kube-api-access-rjsrm\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhdb\" (UniqueName: \"kubernetes.io/projected/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-kube-api-access-7fhdb\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123591    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-log-socket\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123611    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-etc-selinux\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123618    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/001cfcc8-dc86-4070-b7e7-f374fbae8581-hosts-file\") pod \"node-resolver-57dwz\" (UID: \"001cfcc8-dc86-4070-b7e7-f374fbae8581\") " pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123629    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-run-netns\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-sysctl-conf\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123664    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001cfcc8-dc86-4070-b7e7-f374fbae8581-tmp-dir\") pod \"node-resolver-57dwz\" (UID: \"001cfcc8-dc86-4070-b7e7-f374fbae8581\") " pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123686    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-cni-bin\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.125369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqp6\" (UniqueName: \"kubernetes.io/projected/c85e413e-151e-4792-8dfe-18d8b0b07ecf-kube-api-access-lhqp6\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-run-systemd\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123760    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-registration-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123761    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-var-lib-kubelet\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123768    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-systemd\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123797    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-conf-dir\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123805    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb5dw\" (UniqueName: \"kubernetes.io/projected/001cfcc8-dc86-4070-b7e7-f374fbae8581-kube-api-access-rb5dw\") pod \"node-resolver-57dwz\" (UID: \"001cfcc8-dc86-4070-b7e7-f374fbae8581\") " pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123811    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-systemd\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.122882    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d83cf8-33cd-4193-b820-b0b81302862d-host-slash\") pod \"iptables-alerter-dvqr7\" (UID: \"05d83cf8-33cd-4193-b820-b0b81302862d\") " pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123841    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-system-cni-dir\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-run-ovn\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-run-ovn\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124039    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-os-release\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124092    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-socket-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124106    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-system-cni-dir\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124156    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-sys-fs\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124156    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-kubernetes\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124244    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.126395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124336    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-lib-modules\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.124373    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs podName:d14e709b-eb10-4251-80dc-54e1aa91e1dd nodeName:}" failed. No retries permitted until 2026-07-29 10:28:48.624353226 +0000 UTC m=+3.017566083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs") pod "network-metrics-daemon-ddfml" (UID: "d14e709b-eb10-4251-80dc-54e1aa91e1dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124623    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-socket-dir-parent\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124678    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-var-lib-kubelet\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-ovnkube-script-lib\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-cnibin\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125040    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-systemd-units\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125131    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-sysctl-d\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125170    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-run-systemd\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125234    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-etc-openvswitch\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125275    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-var-lib-cni-multus\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb2a6946-777b-4f49-af81-0942eaffea47-etc-tuned\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-log-socket\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-env-overrides\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125859    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/262e51a4-76b2-4a18-9814-9dd42ccf84fd-multus-daemon-config\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125935    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-etc-kubernetes\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.123362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-host-run-ovn-kubernetes\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.125977    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-ovnkube-config\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.128288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.126007    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/262e51a4-76b2-4a18-9814-9dd42ccf84fd-host-run-multus-certs\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.128288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.126008    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/001cfcc8-dc86-4070-b7e7-f374fbae8581-hosts-file\") pod \"node-resolver-57dwz\" (UID: \"001cfcc8-dc86-4070-b7e7-f374fbae8581\") " pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.128288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.126057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-var-lib-openvswitch\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.128288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.126100    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-run-openvswitch\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.128288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.126150    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c85e413e-151e-4792-8dfe-18d8b0b07ecf-device-dir\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.128288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.126528    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05d83cf8-33cd-4193-b820-b0b81302862d-iptables-alerter-script\") pod \"iptables-alerter-dvqr7\" (UID: \"05d83cf8-33cd-4193-b820-b0b81302862d\") " pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.128288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.126542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001cfcc8-dc86-4070-b7e7-f374fbae8581-tmp-dir\") pod \"node-resolver-57dwz\" (UID: \"001cfcc8-dc86-4070-b7e7-f374fbae8581\") " pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.128288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.127297    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb2a6946-777b-4f49-af81-0942eaffea47-tmp\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.128288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.124912    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb2a6946-777b-4f49-af81-0942eaffea47-etc-sysconfig\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.129357 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.129153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-ovn-node-metrics-cert\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.131544 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.131202    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c50ec00e-12f4-41e9-be76-5ab3c1571fad-agent-certs\") pod \"konnectivity-agent-2kzz5\" (UID: \"c50ec00e-12f4-41e9-be76-5ab3c1571fad\") " pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:28:48.133423 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.133398    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:48.133547 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.133429    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:48.133547 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.133444    2569 projected.go:196] Error preparing data for projected volume kube-api-access-r96xc for pod openshift-network-diagnostics/network-check-target-kbgr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:48.133547 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.133518    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc podName:3b538a90-452d-45ff-994c-62c83fa7101b nodeName:}" failed. No retries permitted until 2026-07-29 10:28:48.6334857 +0000 UTC m=+3.026698551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r96xc" (UniqueName: "kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc") pod "network-check-target-kbgr9" (UID: "3b538a90-452d-45ff-994c-62c83fa7101b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:48.134438 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.134411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpn5t\" (UniqueName: \"kubernetes.io/projected/d14e709b-eb10-4251-80dc-54e1aa91e1dd-kube-api-access-rpn5t\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:48.136394 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.136345    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjsrm\" (UniqueName: \"kubernetes.io/projected/262e51a4-76b2-4a18-9814-9dd42ccf84fd-kube-api-access-rjsrm\") pod \"multus-6nndf\" (UID: \"262e51a4-76b2-4a18-9814-9dd42ccf84fd\") " pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.136394 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.136364    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqp6\" (UniqueName: \"kubernetes.io/projected/c85e413e-151e-4792-8dfe-18d8b0b07ecf-kube-api-access-lhqp6\") pod \"aws-ebs-csi-driver-node-4hjns\" (UID: \"c85e413e-151e-4792-8dfe-18d8b0b07ecf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.137879 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.137836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqs5b\" (UniqueName: \"kubernetes.io/projected/05d83cf8-33cd-4193-b820-b0b81302862d-kube-api-access-rqs5b\") pod \"iptables-alerter-dvqr7\" (UID: \"05d83cf8-33cd-4193-b820-b0b81302862d\") " pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.139316 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.139256    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhdb\" (UniqueName: \"kubernetes.io/projected/33c14f37-1dc5-41cc-87c9-c0c3ff9a9400-kube-api-access-7fhdb\") pod \"ovnkube-node-tpvc8\" (UID: \"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400\") " pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.139418 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.139381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb5dw\" (UniqueName: \"kubernetes.io/projected/001cfcc8-dc86-4070-b7e7-f374fbae8581-kube-api-access-rb5dw\") pod \"node-resolver-57dwz\" (UID: \"001cfcc8-dc86-4070-b7e7-f374fbae8581\") " pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.139818 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.139780    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzhx\" (UniqueName: \"kubernetes.io/projected/eb2a6946-777b-4f49-af81-0942eaffea47-kube-api-access-dvzhx\") pod \"tuned-rrkxz\" (UID: \"eb2a6946-777b-4f49-af81-0942eaffea47\") " pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.214682 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.214643    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8gj9q"
Jul 29 10:28:48.222744 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.222710    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-45d2k"
Jul 29 10:28:48.233800 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.233760    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dvqr7"
Jul 29 10:28:48.240603 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.240583    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns"
Jul 29 10:28:48.247692 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.247672    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rrkxz"
Jul 29 10:28:48.255923 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.255733    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57dwz"
Jul 29 10:28:48.264062 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.264036    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6nndf"
Jul 29 10:28:48.271339 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.271177    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:28:48.277339 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.277230    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:28:48.322644 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.322611    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:48.462991 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:48.462954    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262e51a4_76b2_4a18_9814_9dd42ccf84fd.slice/crio-51dad8c284d19849449d5363abc7a445b5e2b4933d08bccbb47d02d8ddcdf04c WatchSource:0}: Error finding container 51dad8c284d19849449d5363abc7a445b5e2b4933d08bccbb47d02d8ddcdf04c: Status 404 returned error can't find the container with id 51dad8c284d19849449d5363abc7a445b5e2b4933d08bccbb47d02d8ddcdf04c
Jul 29 10:28:48.465712 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:48.465680    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50ec00e_12f4_41e9_be76_5ab3c1571fad.slice/crio-9a664310af86c84f3c8705746a20b04cd66fec1c50d433e794dfd65e9f4fe5ca WatchSource:0}: Error finding container 9a664310af86c84f3c8705746a20b04cd66fec1c50d433e794dfd65e9f4fe5ca: Status 404 returned error can't find the container with id 9a664310af86c84f3c8705746a20b04cd66fec1c50d433e794dfd65e9f4fe5ca
Jul 29 10:28:48.491187 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:48.491134    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c14f37_1dc5_41cc_87c9_c0c3ff9a9400.slice/crio-6063f4add62cf24ef90b130b39e727b60c8edfde6ccd5887dc5aaa530ed38c7c WatchSource:0}: Error finding container 6063f4add62cf24ef90b130b39e727b60c8edfde6ccd5887dc5aaa530ed38c7c: Status 404 returned error can't find the container with id 6063f4add62cf24ef90b130b39e727b60c8edfde6ccd5887dc5aaa530ed38c7c
Jul 29 10:28:48.492593 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:48.492562    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2a6946_777b_4f49_af81_0942eaffea47.slice/crio-5339f7dca5646b83d7032a109b4ad5ef680647146ba1171cae100016780e4329 WatchSource:0}: Error finding container 5339f7dca5646b83d7032a109b4ad5ef680647146ba1171cae100016780e4329: Status 404 returned error can't find the container with id 5339f7dca5646b83d7032a109b4ad5ef680647146ba1171cae100016780e4329
Jul 29 10:28:48.493557 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:48.493529    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d83cf8_33cd_4193_b820_b0b81302862d.slice/crio-2efb8f954889d522a67e480d6bb98b42eb6a7a435032d4869714f8379631529c WatchSource:0}: Error finding container 2efb8f954889d522a67e480d6bb98b42eb6a7a435032d4869714f8379631529c: Status 404 returned error can't find the container with id 2efb8f954889d522a67e480d6bb98b42eb6a7a435032d4869714f8379631529c
Jul 29 10:28:48.498264 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:48.498240    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85e413e_151e_4792_8dfe_18d8b0b07ecf.slice/crio-df3f65cbd24898e9711c0a8af2ca1c389ede00ab2506ca2dd6151f8b3c187629 WatchSource:0}: Error finding container df3f65cbd24898e9711c0a8af2ca1c389ede00ab2506ca2dd6151f8b3c187629: Status 404 returned error can't find the container with id df3f65cbd24898e9711c0a8af2ca1c389ede00ab2506ca2dd6151f8b3c187629
Jul 29 10:28:48.499229 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:28:48.499204    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32248a0a_b444_475d_8a21_2139f1dde3fa.slice/crio-d788aa9458b334e5ef09a63c1fcbd0006c6e9bcb654c9fff14e9456b3520bfa9 WatchSource:0}: Error finding container d788aa9458b334e5ef09a63c1fcbd0006c6e9bcb654c9fff14e9456b3520bfa9: Status 404 returned error can't find the container with id d788aa9458b334e5ef09a63c1fcbd0006c6e9bcb654c9fff14e9456b3520bfa9
Jul 29 10:28:48.527129 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.527105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:48.527249 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.527235    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:48.527297 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.527286    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret podName:2285452e-2515-4e83-8aa2-6334577e0140 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:49.527272274 +0000 UTC m=+3.920485124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret") pod "global-pull-secret-syncer-r778m" (UID: "2285452e-2515-4e83-8aa2-6334577e0140") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:48.628461 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.628408    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:48.628646 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.628599    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:48.628691 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.628684    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs podName:d14e709b-eb10-4251-80dc-54e1aa91e1dd nodeName:}" failed. No retries permitted until 2026-07-29 10:28:49.628665434 +0000 UTC m=+4.021878288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs") pod "network-metrics-daemon-ddfml" (UID: "d14e709b-eb10-4251-80dc-54e1aa91e1dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:48.729504 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.729469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:48.729666 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.729644    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:48.729772 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.729671    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:48.729772 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.729683    2569 projected.go:196] Error preparing data for projected volume kube-api-access-r96xc for pod openshift-network-diagnostics/network-check-target-kbgr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:48.729772 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:48.729747    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc podName:3b538a90-452d-45ff-994c-62c83fa7101b nodeName:}" failed. No retries permitted until 2026-07-29 10:28:49.729726818 +0000 UTC m=+4.122939697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r96xc" (UniqueName: "kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc") pod "network-check-target-kbgr9" (UID: "3b538a90-452d-45ff-994c-62c83fa7101b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:48.963769 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.963699    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:48 +0000 UTC" deadline="2028-02-10 03:40:30.317797112 +0000 UTC"
Jul 29 10:28:48.963769 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:48.963749    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13457h11m41.3540515s"
Jul 29 10:28:49.035919 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.035837    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:49.036071 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:49.035990    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:28:49.046288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.046178    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6nndf" event={"ID":"262e51a4-76b2-4a18-9814-9dd42ccf84fd","Type":"ContainerStarted","Data":"51dad8c284d19849449d5363abc7a445b5e2b4933d08bccbb47d02d8ddcdf04c"}
Jul 29 10:28:49.049894 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.049231    2569 generic.go:356] "Generic (PLEG): container finished" podID="0483318e6ca30ead09c5152f256a0232" containerID="62f085d321b219ffd3d86cf136f5361d1fd76ded78ebbd15f0e48347353eb638" exitCode=0
Jul 29 10:28:49.049894 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.049688    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" event={"ID":"0483318e6ca30ead09c5152f256a0232","Type":"ContainerDied","Data":"62f085d321b219ffd3d86cf136f5361d1fd76ded78ebbd15f0e48347353eb638"}
Jul 29 10:28:49.051789 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.051452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45d2k" event={"ID":"32248a0a-b444-475d-8a21-2139f1dde3fa","Type":"ContainerStarted","Data":"d788aa9458b334e5ef09a63c1fcbd0006c6e9bcb654c9fff14e9456b3520bfa9"}
Jul 29 10:28:49.052923 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.052896    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57dwz" event={"ID":"001cfcc8-dc86-4070-b7e7-f374fbae8581","Type":"ContainerStarted","Data":"c446e312de6040287638430742e1d70d0a17288bc9b0ee13d0f13ec3999a0ddc"}
Jul 29 10:28:49.055314 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.055247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns" event={"ID":"c85e413e-151e-4792-8dfe-18d8b0b07ecf","Type":"ContainerStarted","Data":"df3f65cbd24898e9711c0a8af2ca1c389ede00ab2506ca2dd6151f8b3c187629"}
Jul 29 10:28:49.057222 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.057190    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2kzz5" event={"ID":"c50ec00e-12f4-41e9-be76-5ab3c1571fad","Type":"ContainerStarted","Data":"9a664310af86c84f3c8705746a20b04cd66fec1c50d433e794dfd65e9f4fe5ca"}
Jul 29 10:28:49.058899 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.058872    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8gj9q" event={"ID":"a9a64c38-a6c8-4d27-9d16-77605f9f5742","Type":"ContainerStarted","Data":"adf43c43d86ece50053c8c5ab94f8f2849ff510e88c760a945ed2343a45a2fc2"}
Jul 29 10:28:49.060857 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.060798    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dvqr7" event={"ID":"05d83cf8-33cd-4193-b820-b0b81302862d","Type":"ContainerStarted","Data":"2efb8f954889d522a67e480d6bb98b42eb6a7a435032d4869714f8379631529c"}
Jul 29 10:28:49.062101 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.062073    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rrkxz" event={"ID":"eb2a6946-777b-4f49-af81-0942eaffea47","Type":"ContainerStarted","Data":"5339f7dca5646b83d7032a109b4ad5ef680647146ba1171cae100016780e4329"}
Jul 29 10:28:49.063838 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.063813    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" event={"ID":"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400","Type":"ContainerStarted","Data":"6063f4add62cf24ef90b130b39e727b60c8edfde6ccd5887dc5aaa530ed38c7c"}
Jul 29 10:28:49.537727 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.537424    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:49.537727 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:49.537625    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:49.537962 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:49.537796    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret podName:2285452e-2515-4e83-8aa2-6334577e0140 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:51.537775166 +0000 UTC m=+5.930988021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret") pod "global-pull-secret-syncer-r778m" (UID: "2285452e-2515-4e83-8aa2-6334577e0140") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:49.639462 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.638762    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:49.639462 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:49.638927    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:49.639462 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:49.638997    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs podName:d14e709b-eb10-4251-80dc-54e1aa91e1dd nodeName:}" failed. No retries permitted until 2026-07-29 10:28:51.638977563 +0000 UTC m=+6.032190418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs") pod "network-metrics-daemon-ddfml" (UID: "d14e709b-eb10-4251-80dc-54e1aa91e1dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:49.739645 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.739595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:49.739824 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:49.739798    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:49.739824 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:49.739824    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:49.739959 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:49.739840    2569 projected.go:196] Error preparing data for projected volume kube-api-access-r96xc for pod openshift-network-diagnostics/network-check-target-kbgr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:49.739959 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:49.739903    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc podName:3b538a90-452d-45ff-994c-62c83fa7101b nodeName:}" failed. No retries permitted until 2026-07-29 10:28:51.73988344 +0000 UTC m=+6.133096294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r96xc" (UniqueName: "kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc") pod "network-check-target-kbgr9" (UID: "3b538a90-452d-45ff-994c-62c83fa7101b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:49.964837 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.964705    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:48 +0000 UTC" deadline="2028-03-08 13:53:49.042661979 +0000 UTC"
Jul 29 10:28:49.964837 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:49.964752    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14115h24m59.077915753s"
Jul 29 10:28:50.038310 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:50.038217    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:50.038310 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:50.038233    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:50.038871 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:50.038363    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:28:50.038871 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:50.038481    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:28:50.093912 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:50.093609    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" event={"ID":"f186ee96a5b10264dea34206a4faa43e","Type":"ContainerStarted","Data":"6e9d8982f0fe07146fa3f3cafb77ca234499b7a617e98584a103669660c1ffe5"}
Jul 29 10:28:50.103771 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:50.103738    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" event={"ID":"0483318e6ca30ead09c5152f256a0232","Type":"ContainerStarted","Data":"09f86df5f972060a36d11073a847d9b1cce51cb04e3675e3b33f9e350f6da395"}
Jul 29 10:28:50.128397 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:50.128325    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" podStartSLOduration=3.128292577 podStartE2EDuration="3.128292577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:28:50.111566321 +0000 UTC m=+4.504779200" watchObservedRunningTime="2026-07-29 10:28:50.128292577 +0000 UTC m=+4.521505453"
Jul 29 10:28:50.128620 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:50.128478    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" podStartSLOduration=3.128468212 podStartE2EDuration="3.128468212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:28:50.127582078 +0000 UTC m=+4.520794964" watchObservedRunningTime="2026-07-29 10:28:50.128468212 +0000 UTC m=+4.521681087"
Jul 29 10:28:51.036193 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:51.036141    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:51.036698 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:51.036288    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:28:51.555420 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:51.554690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:51.555420 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:51.554951    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:51.555420 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:51.555018    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret podName:2285452e-2515-4e83-8aa2-6334577e0140 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:55.554999404 +0000 UTC m=+9.948212266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret") pod "global-pull-secret-syncer-r778m" (UID: "2285452e-2515-4e83-8aa2-6334577e0140") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:51.655730 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:51.655692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:51.655896 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:51.655843    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:51.655966 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:51.655910    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs podName:d14e709b-eb10-4251-80dc-54e1aa91e1dd nodeName:}" failed. No retries permitted until 2026-07-29 10:28:55.655890103 +0000 UTC m=+10.049102968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs") pod "network-metrics-daemon-ddfml" (UID: "d14e709b-eb10-4251-80dc-54e1aa91e1dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:51.758213 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:51.756901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:51.758213 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:51.757079    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:51.758213 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:51.757097    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:51.758213 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:51.757109    2569 projected.go:196] Error preparing data for projected volume kube-api-access-r96xc for pod openshift-network-diagnostics/network-check-target-kbgr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:51.758213 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:51.757170    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc podName:3b538a90-452d-45ff-994c-62c83fa7101b nodeName:}" failed. No retries permitted until 2026-07-29 10:28:55.757150897 +0000 UTC m=+10.150363759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r96xc" (UniqueName: "kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc") pod "network-check-target-kbgr9" (UID: "3b538a90-452d-45ff-994c-62c83fa7101b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:52.036537 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:52.036387    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:52.036537 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:52.036528    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:28:52.037083 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:52.036907    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:52.037083 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:52.036996    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:28:53.035614 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:53.035511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:53.035793 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:53.035655    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:28:54.036420 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:54.035934    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:54.036420 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:54.036065    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:28:54.037010 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:54.036979    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:54.037157 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:54.037133    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:28:55.036095 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:55.036058    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:55.036287 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:55.036219    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:28:55.593138 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:55.593101    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:55.593651 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:55.593249    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:55.593651 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:55.593324    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret podName:2285452e-2515-4e83-8aa2-6334577e0140 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:03.593302848 +0000 UTC m=+17.986515712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret") pod "global-pull-secret-syncer-r778m" (UID: "2285452e-2515-4e83-8aa2-6334577e0140") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:55.693962 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:55.693858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:55.694129 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:55.693984    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:55.694129 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:55.694052    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs podName:d14e709b-eb10-4251-80dc-54e1aa91e1dd nodeName:}" failed. No retries permitted until 2026-07-29 10:29:03.694034042 +0000 UTC m=+18.087246890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs") pod "network-metrics-daemon-ddfml" (UID: "d14e709b-eb10-4251-80dc-54e1aa91e1dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:55.794907 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:55.794870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:55.795101 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:55.795059    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:55.795101 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:55.795086    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:55.795101 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:55.795100    2569 projected.go:196] Error preparing data for projected volume kube-api-access-r96xc for pod openshift-network-diagnostics/network-check-target-kbgr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:55.795252 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:55.795166    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc podName:3b538a90-452d-45ff-994c-62c83fa7101b nodeName:}" failed. No retries permitted until 2026-07-29 10:29:03.795146359 +0000 UTC m=+18.188359227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r96xc" (UniqueName: "kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc") pod "network-check-target-kbgr9" (UID: "3b538a90-452d-45ff-994c-62c83fa7101b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:56.036951 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:56.036789    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:56.036951 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:56.036924    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:28:56.037326 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:56.037277    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:56.038124 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:56.038063    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:28:57.035992 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:57.035955    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:57.036468 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:57.036102    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:28:58.036397 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:58.036365    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:28:58.036812 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:58.036365    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:28:58.036812 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:58.036510    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:28:58.036812 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:58.036626    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:28:59.035818 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:28:59.035786    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:28:59.035979 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:28:59.035914    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:00.036553 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:00.036522    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:00.036553 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:00.036537    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:00.037007 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:00.036636    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:00.037007 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:00.036716    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:01.035488 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:01.035453    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:01.035648 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:01.035593    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:02.036016 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:02.035984    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:02.036498 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:02.035984    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:02.036498 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:02.036113    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:02.036498 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:02.036200    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:03.036340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:03.036301    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:03.036910 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:03.036465    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:03.657712 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:03.655729    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:03.657712 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:03.655974    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:03.657712 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:03.656059    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret podName:2285452e-2515-4e83-8aa2-6334577e0140 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:19.656039174 +0000 UTC m=+34.049252038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret") pod "global-pull-secret-syncer-r778m" (UID: "2285452e-2515-4e83-8aa2-6334577e0140") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:03.756653 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:03.756611    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:03.756848 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:03.756825    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:03.756929 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:03.756916    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs podName:d14e709b-eb10-4251-80dc-54e1aa91e1dd nodeName:}" failed. No retries permitted until 2026-07-29 10:29:19.756891636 +0000 UTC m=+34.150104505 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs") pod "network-metrics-daemon-ddfml" (UID: "d14e709b-eb10-4251-80dc-54e1aa91e1dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:03.857221 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:03.857185    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:03.857386 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:03.857350    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:03.857386 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:03.857367    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:03.857386 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:03.857377    2569 projected.go:196] Error preparing data for projected volume kube-api-access-r96xc for pod openshift-network-diagnostics/network-check-target-kbgr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:03.857530 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:03.857443    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc podName:3b538a90-452d-45ff-994c-62c83fa7101b nodeName:}" failed. No retries permitted until 2026-07-29 10:29:19.857414443 +0000 UTC m=+34.250627299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r96xc" (UniqueName: "kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc") pod "network-check-target-kbgr9" (UID: "3b538a90-452d-45ff-994c-62c83fa7101b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:04.036406 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:04.036320    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:04.036406 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:04.036354    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:04.036850 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:04.036458    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:04.036850 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:04.036569    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:05.036485 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:05.036451    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:05.036927 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:05.036590    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:06.036388 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:06.036355    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:06.036610 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:06.036471    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:06.036610 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:06.036554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:06.036986 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:06.036671    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:07.036348 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:07.036317    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:07.036527 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:07.036445    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:08.036580 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.036398    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:08.037160 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.036398    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:08.037160 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:08.036674    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:08.037160 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:08.036727    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:08.153732 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.153641    2569 generic.go:356] "Generic (PLEG): container finished" podID="32248a0a-b444-475d-8a21-2139f1dde3fa" containerID="ebff314b0d03fd9830a5039586df6a8687ded497d6eb2876c849081f9aecda7a" exitCode=0
Jul 29 10:29:08.153732 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.153722    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45d2k" event={"ID":"32248a0a-b444-475d-8a21-2139f1dde3fa","Type":"ContainerDied","Data":"ebff314b0d03fd9830a5039586df6a8687ded497d6eb2876c849081f9aecda7a"}
Jul 29 10:29:08.162894 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.162868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57dwz" event={"ID":"001cfcc8-dc86-4070-b7e7-f374fbae8581","Type":"ContainerStarted","Data":"ea856e76495e91eaa01a4857b67ccbdcb6b806b13ed4fb1585c4a8fd4ca15120"}
Jul 29 10:29:08.164584 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.164555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns" event={"ID":"c85e413e-151e-4792-8dfe-18d8b0b07ecf","Type":"ContainerStarted","Data":"eb7235b34298b453a650c2c35e5a0477d263a86da9284dce8b5ec314cbe6f3ac"}
Jul 29 10:29:08.166169 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.166131    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2kzz5" event={"ID":"c50ec00e-12f4-41e9-be76-5ab3c1571fad","Type":"ContainerStarted","Data":"434609f821e5a2cba6b62d0ccbfee84364ed2f17a61b860ecf45c8b75a30e295"}
Jul 29 10:29:08.167852 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.167779    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8gj9q" event={"ID":"a9a64c38-a6c8-4d27-9d16-77605f9f5742","Type":"ContainerStarted","Data":"fdfd8e678d3ccf0139445aa680c15a69fd35867f5bddea97866699be41608f88"}
Jul 29 10:29:08.170287 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.170259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rrkxz" event={"ID":"eb2a6946-777b-4f49-af81-0942eaffea47","Type":"ContainerStarted","Data":"c190657e8df77cb5b5626894fd6c87f90704c0bc98f7a9171a1b6c6c1956e45a"}
Jul 29 10:29:08.174213 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.174189    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" event={"ID":"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400","Type":"ContainerStarted","Data":"37450675e3fa3dcd2a6223baaaff5f8f2f9c97be65e85896539d8a22d5d01b9b"}
Jul 29 10:29:08.174313 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.174263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" event={"ID":"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400","Type":"ContainerStarted","Data":"fc478a2c8829649b43e7dc04169618ee475c69b7a0e1f56ccc92e6155556ac56"}
Jul 29 10:29:08.174378 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.174313    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" event={"ID":"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400","Type":"ContainerStarted","Data":"3112ced2629a17ed9454da6594bbf28cca7908994298b29c2bba4f2c5d7218e0"}
Jul 29 10:29:08.174378 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.174359    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" event={"ID":"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400","Type":"ContainerStarted","Data":"a80ea33a452d302094f9996db05c3ff9179401f4dccb97bfa63fb468d62f6afe"}
Jul 29 10:29:08.174494 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.174406    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" event={"ID":"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400","Type":"ContainerStarted","Data":"03d92b3b567de63537693e987df9d03878990fc18028e26b73a1c91e4157893c"}
Jul 29 10:29:08.174494 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.174471    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" event={"ID":"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400","Type":"ContainerStarted","Data":"d737c4beadbdd49eb95239dda9701658efed0b8ea3a68f67a0d3fd840842415f"}
Jul 29 10:29:08.175736 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.175715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6nndf" event={"ID":"262e51a4-76b2-4a18-9814-9dd42ccf84fd","Type":"ContainerStarted","Data":"493ad25685f2fa03ea9cf9994c34835cdbb0e8414ff305564e43d0c378d5b144"}
Jul 29 10:29:08.224204 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.224096    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2kzz5" podStartSLOduration=3.284518543 podStartE2EDuration="22.224082069s" totalImagesPullingTime="18.939563526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:08.223408285 +0000 UTC m=+22.616621176" watchObservedRunningTime="2026-07-29 10:29:08.224082069 +0000 UTC m=+22.617294940"
Jul 29 10:29:08.224329 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.224251    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6nndf" podStartSLOduration=3.249725393 podStartE2EDuration="22.224246491s" totalImagesPullingTime="18.974521098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:08.202493902 +0000 UTC m=+22.595706775" watchObservedRunningTime="2026-07-29 10:29:08.224246491 +0000 UTC m=+22.617459361"
Jul 29 10:29:08.240139 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.240089    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8gj9q" podStartSLOduration=3.317086652 podStartE2EDuration="22.240074642s" totalImagesPullingTime="18.92298799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:08.239645514 +0000 UTC m=+22.632858385" watchObservedRunningTime="2026-07-29 10:29:08.240074642 +0000 UTC m=+22.633287531"
Jul 29 10:29:08.259998 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.259954    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rrkxz" podStartSLOduration=3.30302824 podStartE2EDuration="22.259942498s" totalImagesPullingTime="18.956914258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:08.259780729 +0000 UTC m=+22.652993600" watchObservedRunningTime="2026-07-29 10:29:08.259942498 +0000 UTC m=+22.653155368"
Jul 29 10:29:08.925793 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.925766    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 10:29:08.984257 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.984125    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T10:29:08.925790041Z","UUID":"68444e83-07a7-46c6-9401-5ca072ac738d","Handler":null,"Name":"","Endpoint":""}
Jul 29 10:29:08.985858 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.985832    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 10:29:08.985986 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:08.985870    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 10:29:09.035908 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:09.035832    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:09.036065 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:09.035998    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:09.180583 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:09.180535    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns" event={"ID":"c85e413e-151e-4792-8dfe-18d8b0b07ecf","Type":"ContainerStarted","Data":"2aa851462f9f1d20d033bfea31ffb73df08e240e64c5f90d6e809e40f9e09677"}
Jul 29 10:29:09.182121 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:09.182091    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dvqr7" event={"ID":"05d83cf8-33cd-4193-b820-b0b81302862d","Type":"ContainerStarted","Data":"4dd5e9781f9f2ed762977a07593b49c8b83a4c1eafbf8c0ee542cae5cde663f6"}
Jul 29 10:29:09.198902 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:09.198856    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-57dwz" podStartSLOduration=4.255798679 podStartE2EDuration="23.198840263s" totalImagesPullingTime="18.943041584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:08.280755695 +0000 UTC m=+22.673968565" watchObservedRunningTime="2026-07-29 10:29:09.198840263 +0000 UTC m=+23.592053178"
Jul 29 10:29:09.199211 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:09.199168    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dvqr7" podStartSLOduration=4.244250667 podStartE2EDuration="23.199158339s" totalImagesPullingTime="18.954907672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:09.198734593 +0000 UTC m=+23.591947463" watchObservedRunningTime="2026-07-29 10:29:09.199158339 +0000 UTC m=+23.592371210"
Jul 29 10:29:10.035771 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:10.035735    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:10.035979 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:10.035896    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:10.035979 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:10.035735    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:10.036093 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:10.036021    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:10.187326 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:10.187248    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns" event={"ID":"c85e413e-151e-4792-8dfe-18d8b0b07ecf","Type":"ContainerStarted","Data":"01d4e1c52a53a19201b3326acd666528e4419f71bdbb340398c44cc4093e73e3"}
Jul 29 10:29:10.208645 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:10.208598    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4hjns" podStartSLOduration=3.07104232 podStartE2EDuration="24.20857859s" totalImagesPullingTime="21.13753627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:10.20792726 +0000 UTC m=+24.601140157" watchObservedRunningTime="2026-07-29 10:29:10.20857859 +0000 UTC m=+24.601791462"
Jul 29 10:29:11.035599 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:11.035569    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:11.035798 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:11.035696    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:11.193446 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:11.193380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" event={"ID":"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400","Type":"ContainerStarted","Data":"145721b163cbd6b31ad32804effbd13df4695ede3a5b1187dc97860c401b07a1"}
Jul 29 10:29:12.036413 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:12.036377    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:12.036612 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:12.036530    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:12.036612 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:12.036582    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:12.036732 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:12.036704    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:12.694285 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:12.694111    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:29:12.694931 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:12.694663    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:29:13.036053 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:13.035983    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:13.036232 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:13.036102    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:13.202338 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:13.202303    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" event={"ID":"33c14f37-1dc5-41cc-87c9-c0c3ff9a9400","Type":"ContainerStarted","Data":"bfb85625d509a447403742c6d9678fb0e75b0ca95206ba68f9b294f563ffb411"}
Jul 29 10:29:13.202572 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:13.202551    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:29:13.204101 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:13.204076    2569 generic.go:356] "Generic (PLEG): container finished" podID="32248a0a-b444-475d-8a21-2139f1dde3fa" containerID="c6cbb9cc2735ee9d7948f2756ca644ee54e3c1ca417558dc755bba467a4811ef" exitCode=0
Jul 29 10:29:13.204236 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:13.204167    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45d2k" event={"ID":"32248a0a-b444-475d-8a21-2139f1dde3fa","Type":"ContainerDied","Data":"c6cbb9cc2735ee9d7948f2756ca644ee54e3c1ca417558dc755bba467a4811ef"}
Jul 29 10:29:13.204402 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:13.204386    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:29:13.204833 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:13.204819    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2kzz5"
Jul 29 10:29:13.218921 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:13.218900    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:29:13.231062 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:13.231018    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8" podStartSLOduration=8.187874868 podStartE2EDuration="27.23100486s" totalImagesPullingTime="19.043129992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:13.230511642 +0000 UTC m=+27.623724512" watchObservedRunningTime="2026-07-29 10:29:13.23100486 +0000 UTC m=+27.624217730"
Jul 29 10:29:14.036253 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.036039    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:14.036763 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.036074    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:14.036763 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:14.036375    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:14.036763 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:14.036389    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:14.207826 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.207796    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:29:14.207955 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.207890    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:29:14.222344 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.222321    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:29:14.661700 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.661656    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ddfml"]
Jul 29 10:29:14.661948 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.661873    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:14.662031 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:14.662004    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:14.662292 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.662271    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r778m"]
Jul 29 10:29:14.662394 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.662365    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:14.662482 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:14.662461    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:14.674364 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.674336    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kbgr9"]
Jul 29 10:29:14.674494 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:14.674455    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:14.674579 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:14.674557    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:15.211214 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:15.211185    2569 generic.go:356] "Generic (PLEG): container finished" podID="32248a0a-b444-475d-8a21-2139f1dde3fa" containerID="a2ac5a75a9d3148c683b2ccf24f7707c2804da902437230aa439fb4ccae641a7" exitCode=0
Jul 29 10:29:15.211661 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:15.211250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45d2k" event={"ID":"32248a0a-b444-475d-8a21-2139f1dde3fa","Type":"ContainerDied","Data":"a2ac5a75a9d3148c683b2ccf24f7707c2804da902437230aa439fb4ccae641a7"}
Jul 29 10:29:16.036333 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:16.036300    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:16.036509 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:16.036392    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:16.036509 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:16.036469    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:16.036588 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:16.036550    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:17.035630 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:17.035583    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:17.036115 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:17.035715    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:17.218065 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:17.218035    2569 generic.go:356] "Generic (PLEG): container finished" podID="32248a0a-b444-475d-8a21-2139f1dde3fa" containerID="358b53b34b5a2e007b581af0ea710e7209f54085573a7d945a490bb0781199f8" exitCode=0
Jul 29 10:29:17.218216 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:17.218089    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45d2k" event={"ID":"32248a0a-b444-475d-8a21-2139f1dde3fa","Type":"ContainerDied","Data":"358b53b34b5a2e007b581af0ea710e7209f54085573a7d945a490bb0781199f8"}
Jul 29 10:29:18.035852 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:18.035615    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:18.036281 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:18.035660    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:18.036281 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:18.035959    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:18.036281 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:18.036041    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:19.036111 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:19.036079    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:19.036494 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:19.036186    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:19.673528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:19.673495    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:19.673773 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:19.673651    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:19.673773 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:19.673724    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret podName:2285452e-2515-4e83-8aa2-6334577e0140 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:51.67370523 +0000 UTC m=+66.066918093 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret") pod "global-pull-secret-syncer-r778m" (UID: "2285452e-2515-4e83-8aa2-6334577e0140") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:19.774220 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:19.774189    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:19.774391 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:19.774358    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:19.774511 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:19.774497    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs podName:d14e709b-eb10-4251-80dc-54e1aa91e1dd nodeName:}" failed. No retries permitted until 2026-07-29 10:29:51.774473799 +0000 UTC m=+66.167686659 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs") pod "network-metrics-daemon-ddfml" (UID: "d14e709b-eb10-4251-80dc-54e1aa91e1dd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:19.875551 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:19.875521    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:19.875723 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:19.875683    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:19.875723 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:19.875701    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:19.875723 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:19.875710    2569 projected.go:196] Error preparing data for projected volume kube-api-access-r96xc for pod openshift-network-diagnostics/network-check-target-kbgr9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:19.875838 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:19.875767    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc podName:3b538a90-452d-45ff-994c-62c83fa7101b nodeName:}" failed. No retries permitted until 2026-07-29 10:29:51.875751736 +0000 UTC m=+66.268964590 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r96xc" (UniqueName: "kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc") pod "network-check-target-kbgr9" (UID: "3b538a90-452d-45ff-994c-62c83fa7101b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:20.036006 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:20.035917    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:20.036163 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:20.035913    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:20.036163 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:20.036096    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ddfml" podUID="d14e709b-eb10-4251-80dc-54e1aa91e1dd"
Jul 29 10:29:20.036581 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:20.036209    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbgr9" podUID="3b538a90-452d-45ff-994c-62c83fa7101b"
Jul 29 10:29:21.035519 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.035471    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:21.035672 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:21.035620    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r778m" podUID="2285452e-2515-4e83-8aa2-6334577e0140"
Jul 29 10:29:21.499962 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.499939    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeReady"
Jul 29 10:29:21.500286 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.500083    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 10:29:21.547914 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.547761    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j5t5h"]
Jul 29 10:29:21.551004 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.550977    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-796bm"]
Jul 29 10:29:21.551109 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.551079    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.553963 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.553942    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-796bm"
Jul 29 10:29:21.554133 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.554110    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rdn4\""
Jul 29 10:29:21.554480 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.554463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 10:29:21.554562 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.554493    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 10:29:21.556785 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.556767    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nxd67\""
Jul 29 10:29:21.556887 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.556791    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 10:29:21.556887 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.556836    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 10:29:21.557065 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.557051    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 10:29:21.565052 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.565030    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j5t5h"]
Jul 29 10:29:21.568414 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.568393    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-796bm"]
Jul 29 10:29:21.687296 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.687264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kwx\" (UniqueName: \"kubernetes.io/projected/8701dd0c-e3d3-4f65-986c-04f2ee51bc34-kube-api-access-d4kwx\") pod \"ingress-canary-796bm\" (UID: \"8701dd0c-e3d3-4f65-986c-04f2ee51bc34\") " pod="openshift-ingress-canary/ingress-canary-796bm"
Jul 29 10:29:21.687420 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.687322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8701dd0c-e3d3-4f65-986c-04f2ee51bc34-cert\") pod \"ingress-canary-796bm\" (UID: \"8701dd0c-e3d3-4f65-986c-04f2ee51bc34\") " pod="openshift-ingress-canary/ingress-canary-796bm"
Jul 29 10:29:21.687420 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.687350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/165e8d7d-5361-458b-91a6-8bea4c4b419a-config-volume\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.687513 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.687419    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.687513 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.687454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/165e8d7d-5361-458b-91a6-8bea4c4b419a-tmp-dir\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.687513 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.687468    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlnzk\" (UniqueName: \"kubernetes.io/projected/165e8d7d-5361-458b-91a6-8bea4c4b419a-kube-api-access-nlnzk\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.788176 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.788141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kwx\" (UniqueName: \"kubernetes.io/projected/8701dd0c-e3d3-4f65-986c-04f2ee51bc34-kube-api-access-d4kwx\") pod \"ingress-canary-796bm\" (UID: \"8701dd0c-e3d3-4f65-986c-04f2ee51bc34\") " pod="openshift-ingress-canary/ingress-canary-796bm"
Jul 29 10:29:21.788317 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.788197    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8701dd0c-e3d3-4f65-986c-04f2ee51bc34-cert\") pod \"ingress-canary-796bm\" (UID: \"8701dd0c-e3d3-4f65-986c-04f2ee51bc34\") " pod="openshift-ingress-canary/ingress-canary-796bm"
Jul 29 10:29:21.788317 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.788226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/165e8d7d-5361-458b-91a6-8bea4c4b419a-config-volume\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.788317 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.788263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.788317 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.788284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/165e8d7d-5361-458b-91a6-8bea4c4b419a-tmp-dir\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.788317 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.788305    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlnzk\" (UniqueName: \"kubernetes.io/projected/165e8d7d-5361-458b-91a6-8bea4c4b419a-kube-api-access-nlnzk\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.788637 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:21.788375    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:21.788637 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:21.788471    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls podName:165e8d7d-5361-458b-91a6-8bea4c4b419a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:22.288451557 +0000 UTC m=+36.681664415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls") pod "dns-default-j5t5h" (UID: "165e8d7d-5361-458b-91a6-8bea4c4b419a") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:21.788778 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.788718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/165e8d7d-5361-458b-91a6-8bea4c4b419a-tmp-dir\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.788880 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.788863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/165e8d7d-5361-458b-91a6-8bea4c4b419a-config-volume\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.792461 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.792442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8701dd0c-e3d3-4f65-986c-04f2ee51bc34-cert\") pod \"ingress-canary-796bm\" (UID: \"8701dd0c-e3d3-4f65-986c-04f2ee51bc34\") " pod="openshift-ingress-canary/ingress-canary-796bm"
Jul 29 10:29:21.797444 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.797403    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlnzk\" (UniqueName: \"kubernetes.io/projected/165e8d7d-5361-458b-91a6-8bea4c4b419a-kube-api-access-nlnzk\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:21.797538 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.797458    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kwx\" (UniqueName: \"kubernetes.io/projected/8701dd0c-e3d3-4f65-986c-04f2ee51bc34-kube-api-access-d4kwx\") pod \"ingress-canary-796bm\" (UID: \"8701dd0c-e3d3-4f65-986c-04f2ee51bc34\") " pod="openshift-ingress-canary/ingress-canary-796bm"
Jul 29 10:29:21.866826 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.866735    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-796bm"
Jul 29 10:29:21.999198 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:21.999169    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-796bm"]
Jul 29 10:29:22.002725 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:29:22.002699    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8701dd0c_e3d3_4f65_986c_04f2ee51bc34.slice/crio-95f36892e326c2d5013e7c22d0e8e7c1c2367f75b178ab5c0dcd471a8d137948 WatchSource:0}: Error finding container 95f36892e326c2d5013e7c22d0e8e7c1c2367f75b178ab5c0dcd471a8d137948: Status 404 returned error can't find the container with id 95f36892e326c2d5013e7c22d0e8e7c1c2367f75b178ab5c0dcd471a8d137948
Jul 29 10:29:22.035858 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.035823    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:22.035858 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.035852    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:22.039237 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.039073    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:22.039371 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.039091    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kqgtv\""
Jul 29 10:29:22.039371 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.039103    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h2t8s\""
Jul 29 10:29:22.039371 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.039139    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:22.039371 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.039154    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:22.230928 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.230845    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-796bm" event={"ID":"8701dd0c-e3d3-4f65-986c-04f2ee51bc34","Type":"ContainerStarted","Data":"95f36892e326c2d5013e7c22d0e8e7c1c2367f75b178ab5c0dcd471a8d137948"}
Jul 29 10:29:22.233513 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.233484    2569 generic.go:356] "Generic (PLEG): container finished" podID="32248a0a-b444-475d-8a21-2139f1dde3fa" containerID="40b9cd00539c8e4aa4fa5723e0d5fd4596270235ff38a46f495178b34cd78a3c" exitCode=0
Jul 29 10:29:22.233617 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.233519    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45d2k" event={"ID":"32248a0a-b444-475d-8a21-2139f1dde3fa","Type":"ContainerDied","Data":"40b9cd00539c8e4aa4fa5723e0d5fd4596270235ff38a46f495178b34cd78a3c"}
Jul 29 10:29:22.292979 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:22.292953    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:22.293123 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:22.293073    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:22.293172 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:22.293135    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls podName:165e8d7d-5361-458b-91a6-8bea4c4b419a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:23.293121145 +0000 UTC m=+37.686333993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls") pod "dns-default-j5t5h" (UID: "165e8d7d-5361-458b-91a6-8bea4c4b419a") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:23.036138 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:23.036102    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:23.039050 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:23.039029    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:23.240197 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:23.240163    2569 generic.go:356] "Generic (PLEG): container finished" podID="32248a0a-b444-475d-8a21-2139f1dde3fa" containerID="6d0a90c12447dbb4db5b2018abf0294c0b46b7dae7928edadb510929c6a040b4" exitCode=0
Jul 29 10:29:23.240361 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:23.240227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45d2k" event={"ID":"32248a0a-b444-475d-8a21-2139f1dde3fa","Type":"ContainerDied","Data":"6d0a90c12447dbb4db5b2018abf0294c0b46b7dae7928edadb510929c6a040b4"}
Jul 29 10:29:23.301371 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:23.301340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:23.301533 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:23.301470    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:23.301533 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:23.301530    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls podName:165e8d7d-5361-458b-91a6-8bea4c4b419a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:25.301511768 +0000 UTC m=+39.694724619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls") pod "dns-default-j5t5h" (UID: "165e8d7d-5361-458b-91a6-8bea4c4b419a") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:23.779675 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:23.779632    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:23.874081 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:23.874030    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:24.245889 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:24.245860    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-45d2k" event={"ID":"32248a0a-b444-475d-8a21-2139f1dde3fa","Type":"ContainerStarted","Data":"45c448ba2c97c9d0bfa8458ad32d925914675862f53e14402cc6621986f8b047"}
Jul 29 10:29:24.270262 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:24.270217    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-45d2k" podStartSLOduration=16.985581941 podStartE2EDuration="38.270200457s" totalImagesPullingTime="21.284618516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:24.268875833 +0000 UTC m=+38.662088706" watchObservedRunningTime="2026-07-29 10:29:24.270200457 +0000 UTC m=+38.663413322"
Jul 29 10:29:24.395403 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:24.395367    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:24.494816 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:24.494781    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:24.565790 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:24.565767    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:24.988021 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:24.987827    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:25.090078 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:25.090045    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:25.249399 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:25.249325    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-796bm" event={"ID":"8701dd0c-e3d3-4f65-986c-04f2ee51bc34","Type":"ContainerStarted","Data":"24d9e7c95641ac6e0c65d50e5c49dc7736609c988b754a4c9d0a262998d4264f"}
Jul 29 10:29:25.266827 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:25.266780    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-796bm" podStartSLOduration=1.8284961819999999 podStartE2EDuration="4.266764679s" totalImagesPullingTime="2.438268497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:25.266050419 +0000 UTC m=+39.659263312" watchObservedRunningTime="2026-07-29 10:29:25.266764679 +0000 UTC m=+39.659977548"
Jul 29 10:29:25.318353 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:25.318326    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:25.318504 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:25.318448    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:25.318504 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:25.318500    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls podName:165e8d7d-5361-458b-91a6-8bea4c4b419a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:29.318485095 +0000 UTC m=+43.711697943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls") pod "dns-default-j5t5h" (UID: "165e8d7d-5361-458b-91a6-8bea4c4b419a") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:29.347239 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:29.347194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:29.347659 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:29.347317    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:29.347659 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:29.347366    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls podName:165e8d7d-5361-458b-91a6-8bea4c4b419a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:37.347353066 +0000 UTC m=+51.740565914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls") pod "dns-default-j5t5h" (UID: "165e8d7d-5361-458b-91a6-8bea4c4b419a") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:31.252916 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:31.252881    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:31.315691 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:31.315651    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:31.598055 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:31.598030    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:31.663888 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:31.663862    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:32.554843 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.554807    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"]
Jul 29 10:29:32.604221 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.604187    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"]
Jul 29 10:29:32.604386 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.604306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:32.607575 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.607553    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 29 10:29:32.607700 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.607631    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:32.608423 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.608404    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 29 10:29:32.608535 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.608477    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g75pl\""
Jul 29 10:29:32.645242 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.645211    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"]
Jul 29 10:29:32.667825 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.667799    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-5dgxd"]
Jul 29 10:29:32.667959 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.667919    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:32.670795 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.670767    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 10:29:32.670929 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.670799    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 29 10:29:32.672097 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.672076    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 10:29:32.672229 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.672110    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 29 10:29:32.672229 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.672121    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wpmdt\""
Jul 29 10:29:32.691887 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.691860    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d6d96c7dc-qc8qb"]
Jul 29 10:29:32.692049 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.691985    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.694685 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.694659    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 10:29:32.694824 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.694729    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cm78f\""
Jul 29 10:29:32.694824 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.694764    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 10:29:32.695028 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.695013    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 10:29:32.695207 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.695191    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 10:29:32.699988 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.699970    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 10:29:32.716167 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.716136    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"]
Jul 29 10:29:32.716167 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.716172    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-5dgxd"]
Jul 29 10:29:32.716341 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.716186    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d6d96c7dc-qc8qb"]
Jul 29 10:29:32.716341 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.716257    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.719625 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.719605    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 29 10:29:32.719747 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.719619    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 29 10:29:32.719747 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.719706    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 29 10:29:32.719747 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.719659    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 10:29:32.720127 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.720106    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 10:29:32.720218 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.720171    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jdrt6\""
Jul 29 10:29:32.720269 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.720232    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 29 10:29:32.771479 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.771445    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:32.771673 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.771497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:32.771673 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.771551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6px4w\" (UniqueName: \"kubernetes.io/projected/54a4a1d6-017e-427a-b461-beed1f1f46bb-kube-api-access-6px4w\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:32.771673 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.771598    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kkh\" (UniqueName: \"kubernetes.io/projected/ff10a068-40c3-427e-bbcc-bc2012d9e43c-kube-api-access-r4kkh\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:32.771785 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.771689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ff10a068-40c3-427e-bbcc-bc2012d9e43c-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:32.831118 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.831082    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:32.872665 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.872626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ff10a068-40c3-427e-bbcc-bc2012d9e43c-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:32.872665 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.872669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2c7d066-1ffc-460b-88c0-709d3f265242-service-ca-bundle\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.872849 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.872722    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:32.872849 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.872739    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2c7d066-1ffc-460b-88c0-709d3f265242-tmp\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.872849 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.872759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2c7d066-1ffc-460b-88c0-709d3f265242-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.872849 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.872781    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-stats-auth\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.872849 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.872817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2c7d066-1ffc-460b-88c0-709d3f265242-serving-cert\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.873032 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:32.872852    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:32.873032 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.872856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.873032 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:32.872918    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls podName:ff10a068-40c3-427e-bbcc-bc2012d9e43c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:33.372904045 +0000 UTC m=+47.766116893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vb4q2" (UID: "ff10a068-40c3-427e-bbcc-bc2012d9e43c") : secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:32.873032 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.872994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:32.873032 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.873022    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6px4w\" (UniqueName: \"kubernetes.io/projected/54a4a1d6-017e-427a-b461-beed1f1f46bb-kube-api-access-6px4w\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:32.873271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.873046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kkh\" (UniqueName: \"kubernetes.io/projected/ff10a068-40c3-427e-bbcc-bc2012d9e43c-kube-api-access-r4kkh\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:32.873271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.873073    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfpcw\" (UniqueName: \"kubernetes.io/projected/c2c7d066-1ffc-460b-88c0-709d3f265242-kube-api-access-lfpcw\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.873271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.873101    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c2c7d066-1ffc-460b-88c0-709d3f265242-snapshots\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.873271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.873126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.873271 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:32.873143    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 10:29:32.873271 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:32.873214    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls podName:54a4a1d6-017e-427a-b461-beed1f1f46bb nodeName:}" failed. No retries permitted until 2026-07-29 10:29:33.373195354 +0000 UTC m=+47.766408203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls") pod "cluster-samples-operator-6748489656-tntpg" (UID: "54a4a1d6-017e-427a-b461-beed1f1f46bb") : secret "samples-operator-tls" not found
Jul 29 10:29:32.873271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.873148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vhq\" (UniqueName: \"kubernetes.io/projected/b299a586-25a6-414c-8717-57184234e96d-kube-api-access-c7vhq\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.873494 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.873274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-default-certificate\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.873527 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.873512    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ff10a068-40c3-427e-bbcc-bc2012d9e43c-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:32.882681 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.882656    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kkh\" (UniqueName: \"kubernetes.io/projected/ff10a068-40c3-427e-bbcc-bc2012d9e43c-kube-api-access-r4kkh\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:32.882830 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.882809    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6px4w\" (UniqueName: \"kubernetes.io/projected/54a4a1d6-017e-427a-b461-beed1f1f46bb-kube-api-access-6px4w\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:32.897344 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.897301    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:32.974611 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.974567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-default-certificate\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.974859 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.974834    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2c7d066-1ffc-460b-88c0-709d3f265242-service-ca-bundle\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.974993 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.974898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2c7d066-1ffc-460b-88c0-709d3f265242-tmp\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.974993 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.974917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2c7d066-1ffc-460b-88c0-709d3f265242-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.974993 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.974940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-stats-auth\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.975160 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975080    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2c7d066-1ffc-460b-88c0-709d3f265242-serving-cert\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.975160 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.975261 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975181    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfpcw\" (UniqueName: \"kubernetes.io/projected/c2c7d066-1ffc-460b-88c0-709d3f265242-kube-api-access-lfpcw\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.975261 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975215    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c2c7d066-1ffc-460b-88c0-709d3f265242-snapshots\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.975261 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.975401 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vhq\" (UniqueName: \"kubernetes.io/projected/b299a586-25a6-414c-8717-57184234e96d-kube-api-access-c7vhq\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.975401 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975303    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2c7d066-1ffc-460b-88c0-709d3f265242-service-ca-bundle\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.975401 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975328    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2c7d066-1ffc-460b-88c0-709d3f265242-tmp\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.975401 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:32.975372    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:33.475348252 +0000 UTC m=+47.868561120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : configmap references non-existent config key: service-ca.crt
Jul 29 10:29:32.975614 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:32.975545    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 10:29:32.975614 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:32.975611    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:33.47559642 +0000 UTC m=+47.868809268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : secret "router-metrics-certs-default" not found
Jul 29 10:29:32.975758 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975735    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2c7d066-1ffc-460b-88c0-709d3f265242-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.975878 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.975858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c2c7d066-1ffc-460b-88c0-709d3f265242-snapshots\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.977263 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.977237    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-default-certificate\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.977508 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.977489    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-stats-auth\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.977646 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.977627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2c7d066-1ffc-460b-88c0-709d3f265242-serving-cert\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:32.983909 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.983887    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vhq\" (UniqueName: \"kubernetes.io/projected/b299a586-25a6-414c-8717-57184234e96d-kube-api-access-c7vhq\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:32.984294 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:32.984278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfpcw\" (UniqueName: \"kubernetes.io/projected/c2c7d066-1ffc-460b-88c0-709d3f265242-kube-api-access-lfpcw\") pod \"insights-operator-5885fdb759-5dgxd\" (UID: \"c2c7d066-1ffc-460b-88c0-709d3f265242\") " pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:33.001132 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:33.001110    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-5dgxd"
Jul 29 10:29:33.157112 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:33.157074    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-5dgxd"]
Jul 29 10:29:33.159626 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:29:33.159603    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c7d066_1ffc_460b_88c0_709d3f265242.slice/crio-ea7cf29e39a66f54d1c041c0f6f9982401004dececff70c3f04bb998be3ac697 WatchSource:0}: Error finding container ea7cf29e39a66f54d1c041c0f6f9982401004dececff70c3f04bb998be3ac697: Status 404 returned error can't find the container with id ea7cf29e39a66f54d1c041c0f6f9982401004dececff70c3f04bb998be3ac697
Jul 29 10:29:33.269172 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:33.269136    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-5dgxd" event={"ID":"c2c7d066-1ffc-460b-88c0-709d3f265242","Type":"ContainerStarted","Data":"ea7cf29e39a66f54d1c041c0f6f9982401004dececff70c3f04bb998be3ac697"}
Jul 29 10:29:33.378136 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:33.378056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:33.378136 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:33.378112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:33.378310 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:33.378183    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:33.378310 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:33.378226    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 10:29:33.378310 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:33.378251    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls podName:ff10a068-40c3-427e-bbcc-bc2012d9e43c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:34.378236095 +0000 UTC m=+48.771448946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vb4q2" (UID: "ff10a068-40c3-427e-bbcc-bc2012d9e43c") : secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:33.378310 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:33.378272    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls podName:54a4a1d6-017e-427a-b461-beed1f1f46bb nodeName:}" failed. No retries permitted until 2026-07-29 10:29:34.378261693 +0000 UTC m=+48.771474540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls") pod "cluster-samples-operator-6748489656-tntpg" (UID: "54a4a1d6-017e-427a-b461-beed1f1f46bb") : secret "samples-operator-tls" not found
Jul 29 10:29:33.479356 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:33.479317    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:33.479544 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:33.479367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:33.479544 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:33.479480    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 10:29:33.479544 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:33.479513    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:34.479495931 +0000 UTC m=+48.872708778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : configmap references non-existent config key: service-ca.crt
Jul 29 10:29:33.479544 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:33.479541    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:34.479531779 +0000 UTC m=+48.872744627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : secret "router-metrics-certs-default" not found
Jul 29 10:29:34.387494 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:34.387457    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:34.387918 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:34.387513    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:34.387918 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:34.387601    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:34.387918 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:34.387664    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls podName:ff10a068-40c3-427e-bbcc-bc2012d9e43c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:36.387648935 +0000 UTC m=+50.780861786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vb4q2" (UID: "ff10a068-40c3-427e-bbcc-bc2012d9e43c") : secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:34.387918 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:34.387607    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 10:29:34.387918 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:34.387728    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls podName:54a4a1d6-017e-427a-b461-beed1f1f46bb nodeName:}" failed. No retries permitted until 2026-07-29 10:29:36.387716915 +0000 UTC m=+50.780929766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls") pod "cluster-samples-operator-6748489656-tntpg" (UID: "54a4a1d6-017e-427a-b461-beed1f1f46bb") : secret "samples-operator-tls" not found
Jul 29 10:29:34.488327 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:34.488288    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:34.488514 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:34.488351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:34.488514 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:34.488472    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 10:29:34.488514 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:34.488474    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:36.488452074 +0000 UTC m=+50.881664922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : configmap references non-existent config key: service-ca.crt
Jul 29 10:29:34.488671 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:34.488533    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:36.488517209 +0000 UTC m=+50.881730060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : secret "router-metrics-certs-default" not found
Jul 29 10:29:35.589502 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:35.589464    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:35.689734 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:35.689687    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:35.790850 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:35.790767    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:36.278045 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.278002    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-5dgxd" event={"ID":"c2c7d066-1ffc-460b-88c0-709d3f265242","Type":"ContainerStarted","Data":"eb5861ed306fa4d776b6c981d1ac99d8232990cf5abe7c33621b23864b87d290"}
Jul 29 10:29:36.295897 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.295849    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-5dgxd" podStartSLOduration=1.995312958 podStartE2EDuration="4.295833398s" totalImagesPullingTime="2.30052044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:36.295439408 +0000 UTC m=+50.688652268" watchObservedRunningTime="2026-07-29 10:29:36.295833398 +0000 UTC m=+50.689046261"
Jul 29 10:29:36.407219 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.404030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:36.407219 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.404167    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:36.407219 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:36.405216    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:36.407219 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:36.405284    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls podName:ff10a068-40c3-427e-bbcc-bc2012d9e43c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.405265349 +0000 UTC m=+54.798478201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vb4q2" (UID: "ff10a068-40c3-427e-bbcc-bc2012d9e43c") : secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:36.407219 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:36.406741    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 10:29:36.407219 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:36.406796    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls podName:54a4a1d6-017e-427a-b461-beed1f1f46bb nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.406779605 +0000 UTC m=+54.799992470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls") pod "cluster-samples-operator-6748489656-tntpg" (UID: "54a4a1d6-017e-427a-b461-beed1f1f46bb") : secret "samples-operator-tls" not found
Jul 29 10:29:36.504716 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.504685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:36.504882 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.504789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:36.504882 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:36.504854    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 10:29:36.505003 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:36.504904    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.504892084 +0000 UTC m=+54.898104932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : configmap references non-existent config key: service-ca.crt
Jul 29 10:29:36.505003 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:36.504934    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.504913453 +0000 UTC m=+54.898126321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : secret "router-metrics-certs-default" not found
Jul 29 10:29:36.892509 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.892472    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2df6f"]
Jul 29 10:29:36.909357 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.909327    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:36.912391 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.912366    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 10:29:36.912846 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.912824    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 10:29:36.912957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.912829    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l6k5g\""
Jul 29 10:29:36.916969 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.916951    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2df6f"]
Jul 29 10:29:36.957035 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:36.956991    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:37.007390 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.007354    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/508e5425-0b4c-4b63-b670-dfb59eade57d-data-volume\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.007578 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.007394    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.007578 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.007455    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9p9p\" (UniqueName: \"kubernetes.io/projected/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-api-access-b9p9p\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.007578 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.007480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/508e5425-0b4c-4b63-b670-dfb59eade57d-crio-socket\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.007578 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.007514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.016039 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.015991    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:37.108558 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.108515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/508e5425-0b4c-4b63-b670-dfb59eade57d-data-volume\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.108558 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.108559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.108779 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.108608    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9p9p\" (UniqueName: \"kubernetes.io/projected/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-api-access-b9p9p\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.108779 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.108636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/508e5425-0b4c-4b63-b670-dfb59eade57d-crio-socket\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.108779 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.108690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.108779 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.108731    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.108928 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.108791    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.108928 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.108798    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:37.608777206 +0000 UTC m=+52.001990069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.108928 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.108825    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:37.608814971 +0000 UTC m=+52.002027818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.108928 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.108849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/508e5425-0b4c-4b63-b670-dfb59eade57d-data-volume\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.108928 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.108857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/508e5425-0b4c-4b63-b670-dfb59eade57d-crio-socket\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.116493 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.116467    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:37.118081 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.118052    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9p9p\" (UniqueName: \"kubernetes.io/projected/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-api-access-b9p9p\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.216763 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.216681    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:37.411794 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.411756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:37.412011 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.411893    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:37.412011 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.411977    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls podName:165e8d7d-5361-458b-91a6-8bea4c4b419a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:53.411957195 +0000 UTC m=+67.805170068 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls") pod "dns-default-j5t5h" (UID: "165e8d7d-5361-458b-91a6-8bea4c4b419a") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:37.613017 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.612980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.613190 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.613065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:37.613190 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.613131    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.613276 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.613202    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:38.613185861 +0000 UTC m=+53.006398708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.613276 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.613136    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.613276 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:37.613276    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:38.613264084 +0000 UTC m=+53.006476931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.813748 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.813713    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:37.916084 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:37.916010    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:38.621272 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:38.621235    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:38.621472 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:38.621319    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:38.621472 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:38.621367    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:38.621472 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:38.621442    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.621416261 +0000 UTC m=+55.014629110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:38.621472 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:38.621458    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:38.621641 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:38.621502    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.621488476 +0000 UTC m=+55.014701324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:38.640095 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:38.640074    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-57dwz_001cfcc8-dc86-4070-b7e7-f374fbae8581/dns-node-resolver/0.log"
Jul 29 10:29:39.214508 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.214475    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:39.303609 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.303577    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:39.370863 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.370834    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:39.412642 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.412612    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-9w82r"]
Jul 29 10:29:39.423349 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.423326    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.426009 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.425982    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-9w82r"]
Jul 29 10:29:39.426226 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.426208    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 10:29:39.426509 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.426492    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 10:29:39.427683 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.427664    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 10:29:39.427769 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.427694    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 10:29:39.427769 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.427668    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9qcp7\""
Jul 29 10:29:39.452784 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.452717    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:39.509453 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.509325    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:39.528581 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.528548    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0a323724-755b-4e8a-af83-784d2482eec3-signing-key\") pod \"service-ca-658d7b7c78-9w82r\" (UID: \"0a323724-755b-4e8a-af83-784d2482eec3\") " pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.528755 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.528737    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a323724-755b-4e8a-af83-784d2482eec3-signing-cabundle\") pod \"service-ca-658d7b7c78-9w82r\" (UID: \"0a323724-755b-4e8a-af83-784d2482eec3\") " pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.528830 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.528769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrkx\" (UniqueName: \"kubernetes.io/projected/0a323724-755b-4e8a-af83-784d2482eec3-kube-api-access-zdrkx\") pod \"service-ca-658d7b7c78-9w82r\" (UID: \"0a323724-755b-4e8a-af83-784d2482eec3\") " pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.629568 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.629534    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0a323724-755b-4e8a-af83-784d2482eec3-signing-key\") pod \"service-ca-658d7b7c78-9w82r\" (UID: \"0a323724-755b-4e8a-af83-784d2482eec3\") " pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.629716 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.629664    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a323724-755b-4e8a-af83-784d2482eec3-signing-cabundle\") pod \"service-ca-658d7b7c78-9w82r\" (UID: \"0a323724-755b-4e8a-af83-784d2482eec3\") " pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.629716 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.629682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrkx\" (UniqueName: \"kubernetes.io/projected/0a323724-755b-4e8a-af83-784d2482eec3-kube-api-access-zdrkx\") pod \"service-ca-658d7b7c78-9w82r\" (UID: \"0a323724-755b-4e8a-af83-784d2482eec3\") " pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.630315 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.630294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a323724-755b-4e8a-af83-784d2482eec3-signing-cabundle\") pod \"service-ca-658d7b7c78-9w82r\" (UID: \"0a323724-755b-4e8a-af83-784d2482eec3\") " pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.631994 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.631970    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0a323724-755b-4e8a-af83-784d2482eec3-signing-key\") pod \"service-ca-658d7b7c78-9w82r\" (UID: \"0a323724-755b-4e8a-af83-784d2482eec3\") " pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.637968 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.637940    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrkx\" (UniqueName: \"kubernetes.io/projected/0a323724-755b-4e8a-af83-784d2482eec3-kube-api-access-zdrkx\") pod \"service-ca-658d7b7c78-9w82r\" (UID: \"0a323724-755b-4e8a-af83-784d2482eec3\") " pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.733743 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.733705    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-9w82r"
Jul 29 10:29:39.840506 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.840370    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8gj9q_a9a64c38-a6c8-4d27-9d16-77605f9f5742/node-ca/0.log"
Jul 29 10:29:39.870974 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:39.870892    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-9w82r"]
Jul 29 10:29:39.873308 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:29:39.873276    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a323724_755b_4e8a_af83_784d2482eec3.slice/crio-5fe8288edb7e1800f424613d675ab1609565bbf84cc3165694191ff4e1aa9536 WatchSource:0}: Error finding container 5fe8288edb7e1800f424613d675ab1609565bbf84cc3165694191ff4e1aa9536: Status 404 returned error can't find the container with id 5fe8288edb7e1800f424613d675ab1609565bbf84cc3165694191ff4e1aa9536
Jul 29 10:29:40.288720 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:40.288678    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-9w82r" event={"ID":"0a323724-755b-4e8a-af83-784d2482eec3","Type":"ContainerStarted","Data":"5fe8288edb7e1800f424613d675ab1609565bbf84cc3165694191ff4e1aa9536"}
Jul 29 10:29:40.307484 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:40.307444    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:40.437413 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:40.437371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:40.437591 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:40.437443    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:40.437591 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.437514    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:40.437591 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.437572    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 10:29:40.437763 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.437593    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls podName:ff10a068-40c3-427e-bbcc-bc2012d9e43c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:48.437573347 +0000 UTC m=+62.830786196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vb4q2" (UID: "ff10a068-40c3-427e-bbcc-bc2012d9e43c") : secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:40.437763 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.437625    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls podName:54a4a1d6-017e-427a-b461-beed1f1f46bb nodeName:}" failed. No retries permitted until 2026-07-29 10:29:48.437607933 +0000 UTC m=+62.830820785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls") pod "cluster-samples-operator-6748489656-tntpg" (UID: "54a4a1d6-017e-427a-b461-beed1f1f46bb") : secret "samples-operator-tls" not found
Jul 29 10:29:40.442254 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:40.442206    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-796bm_8701dd0c-e3d3-4f65-986c-04f2ee51bc34/serve-healthcheck-canary/0.log"
Jul 29 10:29:40.538913 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:40.538824    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:40.539053 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:40.538965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:40.539053 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.539004    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:48.538979583 +0000 UTC m=+62.932192454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : configmap references non-existent config key: service-ca.crt
Jul 29 10:29:40.539149 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.539061    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 10:29:40.539149 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.539119    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:48.539101386 +0000 UTC m=+62.932314256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : secret "router-metrics-certs-default" not found
Jul 29 10:29:40.640302 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:40.640267    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:40.640528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:40.640383    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:40.640528 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.640494    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:40.640528 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.640507    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:40.640664 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.640583    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:44.640560092 +0000 UTC m=+59.033772963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:40.640664 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:40.640607    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:44.640595746 +0000 UTC m=+59.033808610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:42.294917 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:42.294882    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-9w82r" event={"ID":"0a323724-755b-4e8a-af83-784d2482eec3","Type":"ContainerStarted","Data":"b4f7178d6f8ff7c8502bd7089beff8f43b76bdf03ab6da703a0f9fdb5c46531a"}
Jul 29 10:29:42.311804 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:42.311760    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-9w82r" podStartSLOduration=1.569934882 podStartE2EDuration="3.311746035s" totalImagesPullingTime="1.741811153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:42.311342916 +0000 UTC m=+56.704555801" watchObservedRunningTime="2026-07-29 10:29:42.311746035 +0000 UTC m=+56.704958905"
Jul 29 10:29:43.187712 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:43.187681    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:43.351401 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:43.351351    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:43.495132 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:43.495040    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:43.596980 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:43.596936    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:43.733840 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:43.733796    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:43.832056 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:43.832026    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:43.927720 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:43.927687    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:44.675441 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:44.675404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:44.675848 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:44.675563    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:44.675848 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:44.675578    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:44.675848 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:44.675627    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:52.675611617 +0000 UTC m=+67.068824464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:44.675848 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:44.675651    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:44.675848 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:44.675693    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm podName:508e5425-0b4c-4b63-b670-dfb59eade57d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:52.675682545 +0000 UTC m=+67.068895397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2df6f" (UID: "508e5425-0b4c-4b63-b670-dfb59eade57d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:44.880163 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:44.880131    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:45.281736 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:45.281703    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:45.483104 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:45.483059    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:45.682386 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:45.682355    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:45.874157 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:45.874125    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:46.232865 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:46.232840    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tpvc8"
Jul 29 10:29:46.526660 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:46.526576    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:46.875512 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:46.875482    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:46.934013 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:46.933977    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:47.280892 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:47.280801    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:47.382633 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:47.382597    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:47.483974 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:47.483937    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:47.676696 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:47.676664    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:47.878324 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:47.878292    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:47.945695 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:47.945619    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:48.504440 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.504394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:29:48.504615 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.504450    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:48.504615 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:48.504561    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:48.504693 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:48.504646    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls podName:ff10a068-40c3-427e-bbcc-bc2012d9e43c nodeName:}" failed. No retries permitted until 2026-07-29 10:30:04.504624488 +0000 UTC m=+78.897837339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vb4q2" (UID: "ff10a068-40c3-427e-bbcc-bc2012d9e43c") : secret "cluster-monitoring-operator-tls" not found
Jul 29 10:29:48.506848 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.506821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/54a4a1d6-017e-427a-b461-beed1f1f46bb-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tntpg\" (UID: \"54a4a1d6-017e-427a-b461-beed1f1f46bb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:48.516348 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.516321    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g75pl\""
Jul 29 10:29:48.524437 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.524414    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"
Jul 29 10:29:48.605887 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.605858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:48.606139 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.606120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:48.606287 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:29:48.606274    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle podName:b299a586-25a6-414c-8717-57184234e96d nodeName:}" failed. No retries permitted until 2026-07-29 10:30:04.606258752 +0000 UTC m=+78.999471604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle") pod "router-default-d6d96c7dc-qc8qb" (UID: "b299a586-25a6-414c-8717-57184234e96d") : configmap references non-existent config key: service-ca.crt
Jul 29 10:29:48.608262 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.608228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b299a586-25a6-414c-8717-57184234e96d-metrics-certs\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:29:48.646698 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.646671    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg"]
Jul 29 10:29:48.877661 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:48.877614    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:49.275850 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:49.275751    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:49.313436 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:49.313373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg" event={"ID":"54a4a1d6-017e-427a-b461-beed1f1f46bb","Type":"ContainerStarted","Data":"673bd60fa8c6f9bda619029966af9d74f6357d26e5fb93a8ae4dadd1e3d14dad"}
Jul 29 10:29:49.480417 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:49.480371    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:49.577733 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:49.577641    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:49.677626 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:49.677554    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:49.876376 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:49.876347    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:51.073558 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.073525    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:51.272073 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.272021    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:51.319743 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.319711    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg" event={"ID":"54a4a1d6-017e-427a-b461-beed1f1f46bb","Type":"ContainerStarted","Data":"913a6e6d77f756bec0ca80c92d89a2061540fba28d66c158215b0f7fa0ab3fc7"}
Jul 29 10:29:51.319908 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.319760    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg" event={"ID":"54a4a1d6-017e-427a-b461-beed1f1f46bb","Type":"ContainerStarted","Data":"00e01375694bff38d0524c41e5ec3391473f0a05f7b155cdcd264895f314cd44"}
Jul 29 10:29:51.338333 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.338288    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tntpg" podStartSLOduration=17.505899119 podStartE2EDuration="19.338272735s" totalImagesPullingTime="1.832373616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:51.336861395 +0000 UTC m=+65.730074277" watchObservedRunningTime="2026-07-29 10:29:51.338272735 +0000 UTC m=+65.731485632"
Jul 29 10:29:51.476498 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.476470    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:51.577374 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.577341    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:51.677193 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.677106    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:51.732724 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.732693    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:51.735541 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.735520    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:51.745089 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.745067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2285452e-2515-4e83-8aa2-6334577e0140-original-pull-secret\") pod \"global-pull-secret-syncer-r778m\" (UID: \"2285452e-2515-4e83-8aa2-6334577e0140\") " pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:51.833340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.833305    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:51.836033 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.836015    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:51.846583 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.846557    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14e709b-eb10-4251-80dc-54e1aa91e1dd-metrics-certs\") pod \"network-metrics-daemon-ddfml\" (UID: \"d14e709b-eb10-4251-80dc-54e1aa91e1dd\") " pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:51.846708 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.846667    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r778m"
Jul 29 10:29:51.886242 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.886202    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:51.934271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.934238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:51.937565 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.937543    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:51.948736 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.948714    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:51.959268 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.959243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96xc\" (UniqueName: \"kubernetes.io/projected/3b538a90-452d-45ff-994c-62c83fa7101b-kube-api-access-r96xc\") pod \"network-check-target-kbgr9\" (UID: \"3b538a90-452d-45ff-994c-62c83fa7101b\") " pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:51.974964 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:51.974941    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r778m"]
Jul 29 10:29:51.978028 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:29:51.978004    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2285452e_2515_4e83_8aa2_6334577e0140.slice/crio-0f89c52422d461aa68d30732ccc1452854cc26cb9dff105f49c344b1cefc98f3 WatchSource:0}: Error finding container 0f89c52422d461aa68d30732ccc1452854cc26cb9dff105f49c344b1cefc98f3: Status 404 returned error can't find the container with id 0f89c52422d461aa68d30732ccc1452854cc26cb9dff105f49c344b1cefc98f3
Jul 29 10:29:52.049004 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.048972    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h2t8s\""
Jul 29 10:29:52.052342 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.052323    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kqgtv\""
Jul 29 10:29:52.056560 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.056545    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:29:52.060228 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.060215    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ddfml"
Jul 29 10:29:52.189656 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.189542    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kbgr9"]
Jul 29 10:29:52.191986 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:29:52.191957    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b538a90_452d_45ff_994c_62c83fa7101b.slice/crio-186688bf5a0af5c22ceeabafbcaa415ebc702a2df9dc3e897a56b76663985256 WatchSource:0}: Error finding container 186688bf5a0af5c22ceeabafbcaa415ebc702a2df9dc3e897a56b76663985256: Status 404 returned error can't find the container with id 186688bf5a0af5c22ceeabafbcaa415ebc702a2df9dc3e897a56b76663985256
Jul 29 10:29:52.203704 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.203681    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ddfml"]
Jul 29 10:29:52.206288 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:29:52.206265    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14e709b_eb10_4251_80dc_54e1aa91e1dd.slice/crio-8e89ea4c847e39e4b079dcfd6e49b445ec629d789ebfe8ac5dcceea18031daf3 WatchSource:0}: Error finding container 8e89ea4c847e39e4b079dcfd6e49b445ec629d789ebfe8ac5dcceea18031daf3: Status 404 returned error can't find the container with id 8e89ea4c847e39e4b079dcfd6e49b445ec629d789ebfe8ac5dcceea18031daf3
Jul 29 10:29:52.324266 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.324224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kbgr9" event={"ID":"3b538a90-452d-45ff-994c-62c83fa7101b","Type":"ContainerStarted","Data":"186688bf5a0af5c22ceeabafbcaa415ebc702a2df9dc3e897a56b76663985256"}
Jul 29 10:29:52.325796 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.325766    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ddfml" event={"ID":"d14e709b-eb10-4251-80dc-54e1aa91e1dd","Type":"ContainerStarted","Data":"8e89ea4c847e39e4b079dcfd6e49b445ec629d789ebfe8ac5dcceea18031daf3"}
Jul 29 10:29:52.327173 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.327146    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r778m" event={"ID":"2285452e-2515-4e83-8aa2-6334577e0140","Type":"ContainerStarted","Data":"0f89c52422d461aa68d30732ccc1452854cc26cb9dff105f49c344b1cefc98f3"}
Jul 29 10:29:52.741853 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.741811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:52.742078 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.741916    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:52.742927 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.742877    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/508e5425-0b4c-4b63-b670-dfb59eade57d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:52.744972 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.744919    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/508e5425-0b4c-4b63-b670-dfb59eade57d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2df6f\" (UID: \"508e5425-0b4c-4b63-b670-dfb59eade57d\") " pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:52.821534 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.821456    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l6k5g\""
Jul 29 10:29:52.828934 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.828896    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2df6f"
Jul 29 10:29:52.876212 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:52.876178    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:53.005740 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.005602    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2df6f"]
Jul 29 10:29:53.285614 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.285526    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:53.306687 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:29:53.306642    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508e5425_0b4c_4b63_b670_dfb59eade57d.slice/crio-6896b9ec4170855f486778b34bd8afb470e32055b2eefd8fc5f927b3986a970a WatchSource:0}: Error finding container 6896b9ec4170855f486778b34bd8afb470e32055b2eefd8fc5f927b3986a970a: Status 404 returned error can't find the container with id 6896b9ec4170855f486778b34bd8afb470e32055b2eefd8fc5f927b3986a970a
Jul 29 10:29:53.331579 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.331545    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2df6f" event={"ID":"508e5425-0b4c-4b63-b670-dfb59eade57d","Type":"ContainerStarted","Data":"6896b9ec4170855f486778b34bd8afb470e32055b2eefd8fc5f927b3986a970a"}
Jul 29 10:29:53.448156 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.448121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:53.451025 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.450989    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/165e8d7d-5361-458b-91a6-8bea4c4b419a-metrics-tls\") pod \"dns-default-j5t5h\" (UID: \"165e8d7d-5361-458b-91a6-8bea4c4b419a\") " pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:53.484186 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.484152    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:53.664911 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.664884    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rdn4\""
Jul 29 10:29:53.672613 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.672417    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:29:53.682022 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.681979    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:53.858002 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.857870    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j5t5h"]
Jul 29 10:29:53.868124 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:29:53.868089    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165e8d7d_5361_458b_91a6_8bea4c4b419a.slice/crio-f1b1ca05251f451ef1dde5baec3cec325261e7a6cfeac100a8d0782acb006282 WatchSource:0}: Error finding container f1b1ca05251f451ef1dde5baec3cec325261e7a6cfeac100a8d0782acb006282: Status 404 returned error can't find the container with id f1b1ca05251f451ef1dde5baec3cec325261e7a6cfeac100a8d0782acb006282
Jul 29 10:29:53.876301 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:53.876232    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:54.341502 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:54.340545    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ddfml" event={"ID":"d14e709b-eb10-4251-80dc-54e1aa91e1dd","Type":"ContainerStarted","Data":"374daa4b9322a8d7fe9f27b0fa9e7cbbbdf8730383f604cef8b1b2d080a18ccc"}
Jul 29 10:29:54.341502 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:54.340617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ddfml" event={"ID":"d14e709b-eb10-4251-80dc-54e1aa91e1dd","Type":"ContainerStarted","Data":"8ffdaf4ddc2882749947ff6d3cf84bf859873262ffd7ae4e6a43ddd29b5d848f"}
Jul 29 10:29:54.343760 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:54.343687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j5t5h" event={"ID":"165e8d7d-5361-458b-91a6-8bea4c4b419a","Type":"ContainerStarted","Data":"f1b1ca05251f451ef1dde5baec3cec325261e7a6cfeac100a8d0782acb006282"}
Jul 29 10:29:54.359739 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:54.359684    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ddfml" podStartSLOduration=66.935608576 podStartE2EDuration="1m8.359662524s" totalImagesPullingTime="1.424053948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:54.358068918 +0000 UTC m=+68.751281818" watchObservedRunningTime="2026-07-29 10:29:54.359662524 +0000 UTC m=+68.752875397"
Jul 29 10:29:59.120151 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.120111    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"]
Jul 29 10:29:59.144117 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.144086    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"]
Jul 29 10:29:59.144288 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.144201    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.148420 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.148393    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 10:29:59.148568 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.148398    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 10:29:59.148568 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.148442    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 10:29:59.148708 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.148690    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 10:29:59.196781 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.196738    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg578\" (UniqueName: \"kubernetes.io/projected/199d25f1-616e-40c8-b99c-d070cac8831f-kube-api-access-fg578\") pod \"klusterlet-addon-workmgr-5f9cbc4679-4pqbv\" (UID: \"199d25f1-616e-40c8-b99c-d070cac8831f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.196952 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.196828    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/199d25f1-616e-40c8-b99c-d070cac8831f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f9cbc4679-4pqbv\" (UID: \"199d25f1-616e-40c8-b99c-d070cac8831f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.196952 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.196915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/199d25f1-616e-40c8-b99c-d070cac8831f-tmp\") pod \"klusterlet-addon-workmgr-5f9cbc4679-4pqbv\" (UID: \"199d25f1-616e-40c8-b99c-d070cac8831f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.203989 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.203962    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86496c8854-sxqfg"]
Jul 29 10:29:59.208964 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.208934    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.214083 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.213110    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 10:29:59.214083 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.213677    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 10:29:59.214928 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.214709    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8g6k9\""
Jul 29 10:29:59.215671 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.215196    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 10:29:59.220963 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.220939    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 10:29:59.222103 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.222082    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86496c8854-sxqfg"]
Jul 29 10:29:59.230809 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.230774    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:59.297707 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.297671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71bfecff-e8ba-46b8-9d39-a5995d536ce7-ca-trust-extracted\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.297955 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.297739    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/199d25f1-616e-40c8-b99c-d070cac8831f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f9cbc4679-4pqbv\" (UID: \"199d25f1-616e-40c8-b99c-d070cac8831f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.297955 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.297776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8kh8\" (UniqueName: \"kubernetes.io/projected/71bfecff-e8ba-46b8-9d39-a5995d536ce7-kube-api-access-w8kh8\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.297955 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.297802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71bfecff-e8ba-46b8-9d39-a5995d536ce7-bound-sa-token\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.297955 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.297819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71bfecff-e8ba-46b8-9d39-a5995d536ce7-registry-certificates\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.297955 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.297833    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71bfecff-e8ba-46b8-9d39-a5995d536ce7-installation-pull-secrets\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.297955 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.297870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/199d25f1-616e-40c8-b99c-d070cac8831f-tmp\") pod \"klusterlet-addon-workmgr-5f9cbc4679-4pqbv\" (UID: \"199d25f1-616e-40c8-b99c-d070cac8831f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.298296 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.298005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71bfecff-e8ba-46b8-9d39-a5995d536ce7-image-registry-private-configuration\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.298296 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.298038    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71bfecff-e8ba-46b8-9d39-a5995d536ce7-trusted-ca\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.298296 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.298070    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg578\" (UniqueName: \"kubernetes.io/projected/199d25f1-616e-40c8-b99c-d070cac8831f-kube-api-access-fg578\") pod \"klusterlet-addon-workmgr-5f9cbc4679-4pqbv\" (UID: \"199d25f1-616e-40c8-b99c-d070cac8831f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.298296 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.298093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71bfecff-e8ba-46b8-9d39-a5995d536ce7-registry-tls\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.298296 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.298266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/199d25f1-616e-40c8-b99c-d070cac8831f-tmp\") pod \"klusterlet-addon-workmgr-5f9cbc4679-4pqbv\" (UID: \"199d25f1-616e-40c8-b99c-d070cac8831f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.300478 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.300449    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/199d25f1-616e-40c8-b99c-d070cac8831f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5f9cbc4679-4pqbv\" (UID: \"199d25f1-616e-40c8-b99c-d070cac8831f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.334973 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.334936    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg578\" (UniqueName: \"kubernetes.io/projected/199d25f1-616e-40c8-b99c-d070cac8831f-kube-api-access-fg578\") pod \"klusterlet-addon-workmgr-5f9cbc4679-4pqbv\" (UID: \"199d25f1-616e-40c8-b99c-d070cac8831f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.334973 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.334951    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:59.399050 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.398955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71bfecff-e8ba-46b8-9d39-a5995d536ce7-bound-sa-token\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.399050 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.399025    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71bfecff-e8ba-46b8-9d39-a5995d536ce7-registry-certificates\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.399254 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.399055    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71bfecff-e8ba-46b8-9d39-a5995d536ce7-installation-pull-secrets\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.399514 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.399473    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71bfecff-e8ba-46b8-9d39-a5995d536ce7-image-registry-private-configuration\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.399658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.399533    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71bfecff-e8ba-46b8-9d39-a5995d536ce7-trusted-ca\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.399658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.399570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71bfecff-e8ba-46b8-9d39-a5995d536ce7-registry-tls\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.399658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.399606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71bfecff-e8ba-46b8-9d39-a5995d536ce7-ca-trust-extracted\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.399828 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.399668    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8kh8\" (UniqueName: \"kubernetes.io/projected/71bfecff-e8ba-46b8-9d39-a5995d536ce7-kube-api-access-w8kh8\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.400028 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.399979    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71bfecff-e8ba-46b8-9d39-a5995d536ce7-registry-certificates\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.400191 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.400165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71bfecff-e8ba-46b8-9d39-a5995d536ce7-ca-trust-extracted\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.400615 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.400566    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71bfecff-e8ba-46b8-9d39-a5995d536ce7-trusted-ca\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.402057 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.402032    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71bfecff-e8ba-46b8-9d39-a5995d536ce7-installation-pull-secrets\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.402195 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.402173    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71bfecff-e8ba-46b8-9d39-a5995d536ce7-image-registry-private-configuration\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.402516 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.402497    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71bfecff-e8ba-46b8-9d39-a5995d536ce7-registry-tls\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.410358 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.410336    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8kh8\" (UniqueName: \"kubernetes.io/projected/71bfecff-e8ba-46b8-9d39-a5995d536ce7-kube-api-access-w8kh8\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.410653 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.410625    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71bfecff-e8ba-46b8-9d39-a5995d536ce7-bound-sa-token\") pod \"image-registry-86496c8854-sxqfg\" (UID: \"71bfecff-e8ba-46b8-9d39-a5995d536ce7\") " pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.425820 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.425788    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:59.454682 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.454647    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:29:59.522879 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.522841    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:59.523058 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.522998    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:29:59.624949 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.624918    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:59.795617 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.795540    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:29:59.912681 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:29:59.912650    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.025738 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.025698    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.278876 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.278844    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.375254 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.375219    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.478721 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.478677    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.552709 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.552631    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.653793 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.653716    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"]
Jul 29 10:30:00.654332 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.654308    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.658707 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:30:00.658621    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199d25f1_616e_40c8_b99c_d070cac8831f.slice/crio-ae1d3ec3f1467748f1f14bfa6383bfc8fe0f0c98271ad4ba620d959c1c0d2929 WatchSource:0}: Error finding container ae1d3ec3f1467748f1f14bfa6383bfc8fe0f0c98271ad4ba620d959c1c0d2929: Status 404 returned error can't find the container with id ae1d3ec3f1467748f1f14bfa6383bfc8fe0f0c98271ad4ba620d959c1c0d2929
Jul 29 10:30:00.673646 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.673513    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86496c8854-sxqfg"]
Jul 29 10:30:00.676718 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:30:00.676695    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bfecff_e8ba_46b8_9d39_a5995d536ce7.slice/crio-2156c4fa8b7e185ab3b1137a4ce4e649551e8d6b26b0a635903832ce6789c641 WatchSource:0}: Error finding container 2156c4fa8b7e185ab3b1137a4ce4e649551e8d6b26b0a635903832ce6789c641: Status 404 returned error can't find the container with id 2156c4fa8b7e185ab3b1137a4ce4e649551e8d6b26b0a635903832ce6789c641
Jul 29 10:30:00.753004 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.752962    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:00.986053 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:00.986022    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:01.100724 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.100686    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:01.200812 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.200776    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:01.374292 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.374253    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r778m" event={"ID":"2285452e-2515-4e83-8aa2-6334577e0140","Type":"ContainerStarted","Data":"86f5630b4d588b09a02f629c9a4efaedbe39c9d6ef0d2ce73f16066854350b7c"}
Jul 29 10:30:01.377105 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.377009    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j5t5h" event={"ID":"165e8d7d-5361-458b-91a6-8bea4c4b419a","Type":"ContainerStarted","Data":"98eadfd84fa5d131c58d49c6ce7d1f81173e2e07dc4a2058b97648f1339502dd"}
Jul 29 10:30:01.377105 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.377062    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j5t5h" event={"ID":"165e8d7d-5361-458b-91a6-8bea4c4b419a","Type":"ContainerStarted","Data":"ae2b34081af07deba183406ba71dae7f7f1489b9944ff575c2773946f8481aa1"}
Jul 29 10:30:01.377105 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.377092    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:30:01.383390 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.381200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kbgr9" event={"ID":"3b538a90-452d-45ff-994c-62c83fa7101b","Type":"ContainerStarted","Data":"68a2a817c79eb8cc57f38d5834f959abb43ddd5bf050765f521d4fe4cca03135"}
Jul 29 10:30:01.383390 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.381474    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:30:01.383578 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.383476    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv" event={"ID":"199d25f1-616e-40c8-b99c-d070cac8831f","Type":"ContainerStarted","Data":"ae1d3ec3f1467748f1f14bfa6383bfc8fe0f0c98271ad4ba620d959c1c0d2929"}
Jul 29 10:30:01.385945 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.385915    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2df6f" event={"ID":"508e5425-0b4c-4b63-b670-dfb59eade57d","Type":"ContainerStarted","Data":"6fdbaa00011029dffd58449e78092590b5dcc91bc6dda1b5ee3dffd664659a7e"}
Jul 29 10:30:01.387970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.387936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86496c8854-sxqfg" event={"ID":"71bfecff-e8ba-46b8-9d39-a5995d536ce7","Type":"ContainerStarted","Data":"279276f9a722a2849a620b7a516226fb8893822d82e32192f26741225560c589"}
Jul 29 10:30:01.388088 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.387980    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86496c8854-sxqfg" event={"ID":"71bfecff-e8ba-46b8-9d39-a5995d536ce7","Type":"ContainerStarted","Data":"2156c4fa8b7e185ab3b1137a4ce4e649551e8d6b26b0a635903832ce6789c641"}
Jul 29 10:30:01.388317 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.388295    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:30:01.390757 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.390713    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r778m" podStartSLOduration=66.881045601 podStartE2EDuration="1m15.39070139s" totalImagesPullingTime="8.509655789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:01.390120515 +0000 UTC m=+75.783333414" watchObservedRunningTime="2026-07-29 10:30:01.39070139 +0000 UTC m=+75.783914259"
Jul 29 10:30:01.411698 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.411636    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86496c8854-sxqfg" podStartSLOduration=2.411615912 podStartE2EDuration="2.411615912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:01.41006063 +0000 UTC m=+75.803273513" watchObservedRunningTime="2026-07-29 10:30:01.411615912 +0000 UTC m=+75.804828766"
Jul 29 10:30:01.426653 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.425477    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kbgr9" podStartSLOduration=67.125865467 podStartE2EDuration="1m15.425454523s" totalImagesPullingTime="8.299589056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:01.425121129 +0000 UTC m=+75.818333998" watchObservedRunningTime="2026-07-29 10:30:01.425454523 +0000 UTC m=+75.818667387"
Jul 29 10:30:01.476175 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.476132    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:01.713525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.713438    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:01.814896 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:01.814829    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:02.884478 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:02.884396    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:03.324459 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:03.324373    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:04.549944 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:04.549904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:30:04.552655 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:04.552624    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff10a068-40c3-427e-bbcc-bc2012d9e43c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vb4q2\" (UID: \"ff10a068-40c3-427e-bbcc-bc2012d9e43c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:30:04.650800 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:04.650762    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:30:04.651400 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:04.651375    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299a586-25a6-414c-8717-57184234e96d-service-ca-bundle\") pod \"router-default-d6d96c7dc-qc8qb\" (UID: \"b299a586-25a6-414c-8717-57184234e96d\") " pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:30:04.780560 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:04.780525    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wpmdt\""
Jul 29 10:30:04.787892 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:04.787867    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"
Jul 29 10:30:04.827839 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:04.827773    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jdrt6\""
Jul 29 10:30:04.836034 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:04.836008    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:30:05.625381 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:05.625330    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:05.732415 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:05.732371    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:06.483861 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:06.483702    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j5t5h" podStartSLOduration=38.859874308 podStartE2EDuration="45.483677223s" totalImagesPullingTime="6.623802915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:01.446366775 +0000 UTC m=+75.839579647" watchObservedRunningTime="2026-07-29 10:30:06.483677223 +0000 UTC m=+80.876890098"
Jul 29 10:30:06.484495 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:06.484461    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2"]
Jul 29 10:30:06.488456 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:30:06.488406    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff10a068_40c3_427e_bbcc_bc2012d9e43c.slice/crio-fcad28fd6d5f1dca6243932bda27de8b8010e33a7f0e7a75994eaae08a8832b4 WatchSource:0}: Error finding container fcad28fd6d5f1dca6243932bda27de8b8010e33a7f0e7a75994eaae08a8832b4: Status 404 returned error can't find the container with id fcad28fd6d5f1dca6243932bda27de8b8010e33a7f0e7a75994eaae08a8832b4
Jul 29 10:30:06.502721 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:06.502688    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d6d96c7dc-qc8qb"]
Jul 29 10:30:06.506089 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:30:06.506064    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb299a586_25a6_414c_8717_57184234e96d.slice/crio-e948295e2686c8028e69352cd0a68be5784ec01cd83b8a15ad6c8501a52543f4 WatchSource:0}: Error finding container e948295e2686c8028e69352cd0a68be5784ec01cd83b8a15ad6c8501a52543f4: Status 404 returned error can't find the container with id e948295e2686c8028e69352cd0a68be5784ec01cd83b8a15ad6c8501a52543f4
Jul 29 10:30:07.418852 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.418371    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv" event={"ID":"199d25f1-616e-40c8-b99c-d070cac8831f","Type":"ContainerStarted","Data":"78a3d8862f6117ee94d690f9bde8dde888e0c70fccbb52e3ef4160e3e0619ddc"}
Jul 29 10:30:07.418852 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.418550    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:30:07.420213 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.420001    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2" event={"ID":"ff10a068-40c3-427e-bbcc-bc2012d9e43c","Type":"ContainerStarted","Data":"fcad28fd6d5f1dca6243932bda27de8b8010e33a7f0e7a75994eaae08a8832b4"}
Jul 29 10:30:07.420842 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.420821    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv"
Jul 29 10:30:07.422180 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.422067    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d6d96c7dc-qc8qb" event={"ID":"b299a586-25a6-414c-8717-57184234e96d","Type":"ContainerStarted","Data":"c6555bdc0c8939d1a1c4b5a6d2d297b1fdad1ea07a13c23e6713093d9f5693f5"}
Jul 29 10:30:07.422180 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.422114    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d6d96c7dc-qc8qb" event={"ID":"b299a586-25a6-414c-8717-57184234e96d","Type":"ContainerStarted","Data":"e948295e2686c8028e69352cd0a68be5784ec01cd83b8a15ad6c8501a52543f4"}
Jul 29 10:30:07.424077 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.424055    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2df6f" event={"ID":"508e5425-0b4c-4b63-b670-dfb59eade57d","Type":"ContainerStarted","Data":"10fdff8e465e55d01a256785adedd4d057f0f7b6140d2749cdb0ffc0e11e320c"}
Jul 29 10:30:07.435757 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.435705    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5f9cbc4679-4pqbv" podStartSLOduration=2.746429565 podStartE2EDuration="8.435686225s" totalImagesPullingTime="5.68925666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:07.435272595 +0000 UTC m=+81.828485470" watchObservedRunningTime="2026-07-29 10:30:07.435686225 +0000 UTC m=+81.828899088"
Jul 29 10:30:07.454235 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.454183    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-d6d96c7dc-qc8qb" podStartSLOduration=35.454173253 podStartE2EDuration="35.454173253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:07.453926623 +0000 UTC m=+81.847139493" watchObservedRunningTime="2026-07-29 10:30:07.454173253 +0000 UTC m=+81.847386123"
Jul 29 10:30:07.497621 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.497585    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:07.649875 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.649840    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:07.718325 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.718198    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:07.837025 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.836992    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:30:07.840449 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:07.840401    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:30:08.431701 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:08.431605    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2" event={"ID":"ff10a068-40c3-427e-bbcc-bc2012d9e43c","Type":"ContainerStarted","Data":"a696055be1ddd26a85995678f59bfb06a0719bb67576bc6c4d73b603609e3099"}
Jul 29 10:30:08.432153 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:08.432128    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:30:08.434070 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:08.434046    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d6d96c7dc-qc8qb"
Jul 29 10:30:08.450914 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:08.450854    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vb4q2" podStartSLOduration=34.761123469 podStartE2EDuration="36.450832821s" totalImagesPullingTime="1.689709352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:08.448587913 +0000 UTC m=+82.841800784" watchObservedRunningTime="2026-07-29 10:30:08.450832821 +0000 UTC m=+82.844045691"
Jul 29 10:30:08.517296 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:08.517247    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:08.776944 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:08.776849    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:08.848488 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:08.848447    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:09.174641 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:09.174605    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:09.886522 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:09.886489    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:10.440039 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:10.440003    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2df6f" event={"ID":"508e5425-0b4c-4b63-b670-dfb59eade57d","Type":"ContainerStarted","Data":"50467041aa7f15e08cbe852a8919b4d48f8914291381c15b41f253cf814fb915"}
Jul 29 10:30:10.480137 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:10.480084    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2df6f" podStartSLOduration=17.920750282 podStartE2EDuration="34.480066302s" totalImagesPullingTime="16.55931602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:10.47874322 +0000 UTC m=+84.871956102" watchObservedRunningTime="2026-07-29 10:30:10.480066302 +0000 UTC m=+84.873279172"
Jul 29 10:30:10.942997 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:10.942962    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:11.002706 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:11.002677    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:11.395873 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:11.395847    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j5t5h"
Jul 29 10:30:13.711704 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:13.711671    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:13.813660 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:13.813623    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:13.912957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:13.912924    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:13.982323 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:13.982237    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:14.868737 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:14.868695    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:14.926268 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:14.926230    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:16.493670 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.493637    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f96db74d-6bljz"]
Jul 29 10:30:16.496402 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.496379    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.497008 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.496971    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:16.500318 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.500295    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 10:30:16.500318 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.500311    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-k4brd\""
Jul 29 10:30:16.500318 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.500321    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 10:30:16.501060 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.500784    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 10:30:16.501060 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.500899    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 10:30:16.501060 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.500900    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 10:30:16.501060 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.500959    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 10:30:16.503454 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.503411    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 10:30:16.508395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.508371    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 10:30:16.508601 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.508577    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f96db74d-6bljz"]
Jul 29 10:30:16.642220 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.642183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-serving-cert\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.642220 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.642217    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdds4\" (UniqueName: \"kubernetes.io/projected/7529d802-ffcb-430f-aa31-e934ff07c894-kube-api-access-qdds4\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.642455 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.642242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-console-config\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.642455 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.642292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-oauth-serving-cert\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.642455 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.642309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-trusted-ca-bundle\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.642455 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.642364    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-oauth-config\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.642455 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.642415    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-service-ca\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.743359 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.743325    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-serving-cert\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.743359 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.743358    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdds4\" (UniqueName: \"kubernetes.io/projected/7529d802-ffcb-430f-aa31-e934ff07c894-kube-api-access-qdds4\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.743622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.743382    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-console-config\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.743622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.743405    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-oauth-serving-cert\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.743622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.743420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-trusted-ca-bundle\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.743622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.743449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-oauth-config\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.743622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.743474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-service-ca\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.744292 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.744231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-service-ca\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.744397 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.744233    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-oauth-serving-cert\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.744397 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.744272    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-console-config\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.744397 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.744348    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-trusted-ca-bundle\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.746068 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.746047    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-oauth-config\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.746122 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.746053    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-serving-cert\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.752641 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.752620    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdds4\" (UniqueName: \"kubernetes.io/projected/7529d802-ffcb-430f-aa31-e934ff07c894-kube-api-access-qdds4\") pod \"console-67f96db74d-6bljz\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") " pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.808673 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.808638    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:16.938864 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:16.938716    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f96db74d-6bljz"]
Jul 29 10:30:16.941352 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:30:16.941321    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7529d802_ffcb_430f_aa31_e934ff07c894.slice/crio-d718877f3d62f984a1f9c13f7f9b325d5bd1af99eb25af56ee8d32c403a36274 WatchSource:0}: Error finding container d718877f3d62f984a1f9c13f7f9b325d5bd1af99eb25af56ee8d32c403a36274: Status 404 returned error can't find the container with id d718877f3d62f984a1f9c13f7f9b325d5bd1af99eb25af56ee8d32c403a36274
Jul 29 10:30:17.462623 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:17.462579    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f96db74d-6bljz" event={"ID":"7529d802-ffcb-430f-aa31-e934ff07c894","Type":"ContainerStarted","Data":"d718877f3d62f984a1f9c13f7f9b325d5bd1af99eb25af56ee8d32c403a36274"}
Jul 29 10:30:17.499565 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:17.499528    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:17.601055 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:17.600971    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:20.477650 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:20.477613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f96db74d-6bljz" event={"ID":"7529d802-ffcb-430f-aa31-e934ff07c894","Type":"ContainerStarted","Data":"5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6"}
Jul 29 10:30:20.496004 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:20.495951    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67f96db74d-6bljz" podStartSLOduration=1.530410864 podStartE2EDuration="4.495934564s" totalImagesPullingTime="2.9655237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:20.495299048 +0000 UTC m=+94.888511930" watchObservedRunningTime="2026-07-29 10:30:20.495934564 +0000 UTC m=+94.889147434"
Jul 29 10:30:20.638678 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:20.638646    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:20.731574 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:20.731489    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:21.500133 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:21.500086    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:22.397751 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:22.397725    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86496c8854-sxqfg"
Jul 29 10:30:22.441693 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:22.441659    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:22.546290 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:22.546250    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:22.606202 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:22.606167    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:22.737475 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:22.737383    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:22.828531 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:22.828496    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:23.299319 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:23.299286    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:23.399659 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:23.399628    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:23.879446 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:23.879405    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:23.936340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:23.936303    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:24.844637 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:24.844610    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:25.058029 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.057987    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:25.102622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.102529    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"]
Jul 29 10:30:25.108998 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.108973    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.112261 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.112237    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 10:30:25.112612 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.112313    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 29 10:30:25.112770 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.112407    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gbvdj\""
Jul 29 10:30:25.113758 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.113739    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 29 10:30:25.115369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.114421    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 29 10:30:25.115903 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.115885    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tpnck"]
Jul 29 10:30:25.120000 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.119978    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"]
Jul 29 10:30:25.120142 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.120114    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.123469 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.123447    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 10:30:25.123678 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.123471    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 10:30:25.123755 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.123474    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 10:30:25.123833 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.123543    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nhtns\""
Jul 29 10:30:25.167373 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.167345    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:25.213633 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.213599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c39967-9e25-46e3-88dd-d6f07e0559d8-sys\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.213834 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.213640    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8lm\" (UniqueName: \"kubernetes.io/projected/e6c39967-9e25-46e3-88dd-d6f07e0559d8-kube-api-access-dx8lm\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.213834 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.213665    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.213834 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.213785    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6c39967-9e25-46e3-88dd-d6f07e0559d8-metrics-client-ca\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.214011 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.213835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-tls\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.214011 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.213878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjln9\" (UniqueName: \"kubernetes.io/projected/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-api-access-fjln9\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.214011 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.213927    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.214011 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.213961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-textfile\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.214011 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.213986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.214262 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.214019    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.214262 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.214056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-wtmp\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.214385 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.214362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.214524 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.214454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6c39967-9e25-46e3-88dd-d6f07e0559d8-root\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.214524 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.214507    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.214617 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.214542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.301223 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.301190    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:25.315401 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6c39967-9e25-46e3-88dd-d6f07e0559d8-metrics-client-ca\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.315401 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-tls\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.315640 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315454    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjln9\" (UniqueName: \"kubernetes.io/projected/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-api-access-fjln9\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.315640 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.315640 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315527    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-textfile\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.315640 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.315640 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315591    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.315640 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-wtmp\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.315939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.315939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315693    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6c39967-9e25-46e3-88dd-d6f07e0559d8-root\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.315939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315725    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.315939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315753    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.315939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315782    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c39967-9e25-46e3-88dd-d6f07e0559d8-sys\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.315939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8lm\" (UniqueName: \"kubernetes.io/projected/e6c39967-9e25-46e3-88dd-d6f07e0559d8-kube-api-access-dx8lm\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.315939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.315837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.316278 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.316135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6c39967-9e25-46e3-88dd-d6f07e0559d8-metrics-client-ca\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.316278 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.316207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6c39967-9e25-46e3-88dd-d6f07e0559d8-root\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.316278 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.316220    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-wtmp\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.316278 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.316258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c39967-9e25-46e3-88dd-d6f07e0559d8-sys\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.316525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.316474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-textfile\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.316525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.316513    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.317013 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.316990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.317183 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.317150    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.317477 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.317457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.318555 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.318528    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.318672 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.318586    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.318724 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.318708    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6c39967-9e25-46e3-88dd-d6f07e0559d8-node-exporter-tls\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.319251 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.319222    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.323895 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.323871    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjln9\" (UniqueName: \"kubernetes.io/projected/0bd6173a-0f30-42cd-bf68-1dc9a65782ed-kube-api-access-fjln9\") pod \"kube-state-metrics-56d9f944b4-6ksth\" (UID: \"0bd6173a-0f30-42cd-bf68-1dc9a65782ed\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.323999 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.323892    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8lm\" (UniqueName: \"kubernetes.io/projected/e6c39967-9e25-46e3-88dd-d6f07e0559d8-kube-api-access-dx8lm\") pod \"node-exporter-tpnck\" (UID: \"e6c39967-9e25-46e3-88dd-d6f07e0559d8\") " pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.401213 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.401118    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:25.421660 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.421625    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"
Jul 29 10:30:25.431704 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.431677    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tpnck"
Jul 29 10:30:25.444234 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:30:25.444207    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c39967_9e25_46e3_88dd_d6f07e0559d8.slice/crio-71b09ace0c814792e5350b51194f328de4de373b2c5dbc5ce7f6c9a96349ef44 WatchSource:0}: Error finding container 71b09ace0c814792e5350b51194f328de4de373b2c5dbc5ce7f6c9a96349ef44: Status 404 returned error can't find the container with id 71b09ace0c814792e5350b51194f328de4de373b2c5dbc5ce7f6c9a96349ef44
Jul 29 10:30:25.500532 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.500493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tpnck" event={"ID":"e6c39967-9e25-46e3-88dd-d6f07e0559d8","Type":"ContainerStarted","Data":"71b09ace0c814792e5350b51194f328de4de373b2c5dbc5ce7f6c9a96349ef44"}
Jul 29 10:30:25.557012 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.556986    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth"]
Jul 29 10:30:25.559022 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:30:25.558998    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd6173a_0f30_42cd_bf68_1dc9a65782ed.slice/crio-1b78b0293e18ecf4067cb31d9a8c41b80975b8539fc57c8b7481209a072f1ccd WatchSource:0}: Error finding container 1b78b0293e18ecf4067cb31d9a8c41b80975b8539fc57c8b7481209a072f1ccd: Status 404 returned error can't find the container with id 1b78b0293e18ecf4067cb31d9a8c41b80975b8539fc57c8b7481209a072f1ccd
Jul 29 10:30:25.730089 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.729982    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:25.792189 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:25.792143    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:26.076885 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:26.076747    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:26.473846 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:26.473811    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:26.507712 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:26.507665    2569 generic.go:356] "Generic (PLEG): container finished" podID="e6c39967-9e25-46e3-88dd-d6f07e0559d8" containerID="9a1124a793070999f5509983472fe6fd1ad400b1f0319c06a3c5c6203d3a9737" exitCode=0
Jul 29 10:30:26.507908 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:26.507754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tpnck" event={"ID":"e6c39967-9e25-46e3-88dd-d6f07e0559d8","Type":"ContainerDied","Data":"9a1124a793070999f5509983472fe6fd1ad400b1f0319c06a3c5c6203d3a9737"}
Jul 29 10:30:26.509228 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:26.509201    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth" event={"ID":"0bd6173a-0f30-42cd-bf68-1dc9a65782ed","Type":"ContainerStarted","Data":"1b78b0293e18ecf4067cb31d9a8c41b80975b8539fc57c8b7481209a072f1ccd"}
Jul 29 10:30:26.538389 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:26.538320    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:26.809679 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:26.809642    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:26.809874 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:26.809720    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:26.815543 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:26.815512    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:27.517934 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:27.517889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tpnck" event={"ID":"e6c39967-9e25-46e3-88dd-d6f07e0559d8","Type":"ContainerStarted","Data":"da6b1e6f1ffb8ead6a02f591053f8117690cd72d81d2c50470a351015c1d9569"}
Jul 29 10:30:27.518322 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:27.517978    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tpnck" event={"ID":"e6c39967-9e25-46e3-88dd-d6f07e0559d8","Type":"ContainerStarted","Data":"7bffd33a5e39539999f340a87ea34ef8bb0033e2e2fcca69f5ea1dab5dafb50d"}
Jul 29 10:30:27.520024 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:27.519993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth" event={"ID":"0bd6173a-0f30-42cd-bf68-1dc9a65782ed","Type":"ContainerStarted","Data":"a36cd1f2cb913d87d37e3737f1577b12e3969f98c99efb15f12748acd6c08555"}
Jul 29 10:30:27.520156 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:27.520057    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth" event={"ID":"0bd6173a-0f30-42cd-bf68-1dc9a65782ed","Type":"ContainerStarted","Data":"7ad561b2ee2196292c33cc64a7707c52e08f5b181f725f7548c3bb99dbca76dc"}
Jul 29 10:30:27.524475 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:27.524455    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:30:27.541917 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:27.541848    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tpnck" podStartSLOduration=1.643938135 podStartE2EDuration="2.541826822s" totalImagesPullingTime="833.618758ms" totalInitContainerRuntime="64.269929ms" isStatefulPod=true podCreationTimestamp="2026-07-29 10:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:27.54011763 +0000 UTC m=+101.933330506" watchObservedRunningTime="2026-07-29 10:30:27.541826822 +0000 UTC m=+101.935039694"
Jul 29 10:30:28.051667 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.051637    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:28.118486 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.118456    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"]
Jul 29 10:30:28.122685 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.122665    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.125688 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.125648    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ljd24\""
Jul 29 10:30:28.125813 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.125712    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 29 10:30:28.125948 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.125933    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cct54bicieoi6\""
Jul 29 10:30:28.126660 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.126641    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 29 10:30:28.126660 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.126657    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 29 10:30:28.126802 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.126648    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 29 10:30:28.128708 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.128689    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 29 10:30:28.138495 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.138471    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"]
Jul 29 10:30:28.168484 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.168415    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:28.240527 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.240487    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzvqs\" (UniqueName: \"kubernetes.io/projected/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-kube-api-access-xzvqs\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.240719 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.240540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.240719 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.240581    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.240719 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.240609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-grpc-tls\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.240865 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.240708    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.240865 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.240760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-metrics-client-ca\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.240865 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.240843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-tls\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.240985 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.240879    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.341704 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.341665    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-tls\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.341704 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.341709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.341965 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.341741    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzvqs\" (UniqueName: \"kubernetes.io/projected/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-kube-api-access-xzvqs\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.341965 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.341766    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.341965 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.341799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.341965 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.341827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-grpc-tls\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.341965 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.341879    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.341965 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.341907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-metrics-client-ca\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.342714 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.342689    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-metrics-client-ca\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.344943 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.344917    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.345592 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.345057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-grpc-tls\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.345592 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.345458    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.345592 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.345540    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.345736 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.345614    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-tls\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.345736 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.345618    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.350974 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.350952    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzvqs\" (UniqueName: \"kubernetes.io/projected/9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb-kube-api-access-xzvqs\") pod \"thanos-querier-bbf8fcf7d-wk55s\" (UID: \"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb\") " pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.432640 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.432603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:28.526728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.526696    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth" event={"ID":"0bd6173a-0f30-42cd-bf68-1dc9a65782ed","Type":"ContainerStarted","Data":"d06f05296924119753d0cffbca81827135d39bc741e48c40e71c57bcf8ebb1ea"}
Jul 29 10:30:28.554733 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.550043    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-6ksth" podStartSLOduration=1.7770406460000001 podStartE2EDuration="3.550021228s" totalImagesPullingTime="1.772980582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:28.548834656 +0000 UTC m=+102.942047549" watchObservedRunningTime="2026-07-29 10:30:28.550021228 +0000 UTC m=+102.943234098"
Jul 29 10:30:28.582485 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.582242    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"]
Jul 29 10:30:28.587558 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.587526    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:28.588036 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:30:28.588014    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd4be9e_2d45_4e19_a6ef_f8e94389b6cb.slice/crio-96946695d157f38cfc2fac7bf63a0c7d48262682e773b713ce888a2146aa5960 WatchSource:0}: Error finding container 96946695d157f38cfc2fac7bf63a0c7d48262682e773b713ce888a2146aa5960: Status 404 returned error can't find the container with id 96946695d157f38cfc2fac7bf63a0c7d48262682e773b713ce888a2146aa5960
Jul 29 10:30:28.721420 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:28.721386    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:29.432448 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.432395    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"]
Jul 29 10:30:29.452407 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.452375    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"]
Jul 29 10:30:29.452620 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.452519    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.457050 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.457024    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 29 10:30:29.457196 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.457046    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 29 10:30:29.457196 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.457027    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 10:30:29.457196 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.457086    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 29 10:30:29.457196 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.457056    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4jnvq\""
Jul 29 10:30:29.457406 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.457214    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b0jk6q75duucq\""
Jul 29 10:30:29.508377 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.508342    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:29.533891 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.533856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s" event={"ID":"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb","Type":"ContainerStarted","Data":"96946695d157f38cfc2fac7bf63a0c7d48262682e773b713ce888a2146aa5960"}
Jul 29 10:30:29.552656 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.552614    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxsg\" (UniqueName: \"kubernetes.io/projected/f553caa1-3de3-4749-aba5-eed1765f3819-kube-api-access-rcxsg\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.552856 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.552679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f553caa1-3de3-4749-aba5-eed1765f3819-secret-metrics-server-client-certs\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.552856 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.552752    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f553caa1-3de3-4749-aba5-eed1765f3819-secret-metrics-server-tls\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.552856 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.552822    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f553caa1-3de3-4749-aba5-eed1765f3819-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.552856 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.552845    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f553caa1-3de3-4749-aba5-eed1765f3819-audit-log\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.553068 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.552905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f553caa1-3de3-4749-aba5-eed1765f3819-metrics-server-audit-profiles\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.553068 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.552930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f553caa1-3de3-4749-aba5-eed1765f3819-client-ca-bundle\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.568546 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.568512    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:29.653739 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.653695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f553caa1-3de3-4749-aba5-eed1765f3819-secret-metrics-server-client-certs\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.653936 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.653750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f553caa1-3de3-4749-aba5-eed1765f3819-secret-metrics-server-tls\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.653936 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.653813    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f553caa1-3de3-4749-aba5-eed1765f3819-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.653936 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.653833    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f553caa1-3de3-4749-aba5-eed1765f3819-audit-log\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.653936 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.653872    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f553caa1-3de3-4749-aba5-eed1765f3819-metrics-server-audit-profiles\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.653936 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.653910    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f553caa1-3de3-4749-aba5-eed1765f3819-client-ca-bundle\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.654185 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.653976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxsg\" (UniqueName: \"kubernetes.io/projected/f553caa1-3de3-4749-aba5-eed1765f3819-kube-api-access-rcxsg\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.654670 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.654642    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f553caa1-3de3-4749-aba5-eed1765f3819-audit-log\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.654909 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.654883    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f553caa1-3de3-4749-aba5-eed1765f3819-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.655551 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.655515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f553caa1-3de3-4749-aba5-eed1765f3819-metrics-server-audit-profiles\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.656714 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.656691    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f553caa1-3de3-4749-aba5-eed1765f3819-secret-metrics-server-client-certs\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.656873 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.656851    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f553caa1-3de3-4749-aba5-eed1765f3819-client-ca-bundle\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.657104 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.657086    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f553caa1-3de3-4749-aba5-eed1765f3819-secret-metrics-server-tls\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.662107 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.662090    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxsg\" (UniqueName: \"kubernetes.io/projected/f553caa1-3de3-4749-aba5-eed1765f3819-kube-api-access-rcxsg\") pod \"metrics-server-7fb54f9c7d-2hmrd\" (UID: \"f553caa1-3de3-4749-aba5-eed1765f3819\") " pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.766547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.766451    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:29.878066 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.877967    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:29.954024 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.953842    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"]
Jul 29 10:30:29.959488 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:30:29.959452    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf553caa1_3de3_4749_aba5_eed1765f3819.slice/crio-b0f8c06b8ecbf3b6b50ee0393639a6b72e99e151af2c05297452bc98d46825d5 WatchSource:0}: Error finding container b0f8c06b8ecbf3b6b50ee0393639a6b72e99e151af2c05297452bc98d46825d5: Status 404 returned error can't find the container with id b0f8c06b8ecbf3b6b50ee0393639a6b72e99e151af2c05297452bc98d46825d5
Jul 29 10:30:29.971162 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:29.971135    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:30.539571 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:30.539525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd" event={"ID":"f553caa1-3de3-4749-aba5-eed1765f3819","Type":"ContainerStarted","Data":"b0f8c06b8ecbf3b6b50ee0393639a6b72e99e151af2c05297452bc98d46825d5"}
Jul 29 10:30:31.119334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:31.119301    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:31.185314 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:31.185275    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:31.313550 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:31.313402    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:31.545777 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:31.545698    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s" event={"ID":"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb","Type":"ContainerStarted","Data":"9f67bbf6827f5eef40d724e5a3a1c8804aa2a9a651b2c09f3fdeaa06cd3aecc4"}
Jul 29 10:30:31.545777 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:31.545767    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s" event={"ID":"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb","Type":"ContainerStarted","Data":"e55a6aec8ffb88ce144b411bd2f55a9ca10d85ca1cd79a62576f4d56108f8f4a"}
Jul 29 10:30:31.546174 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:31.545795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s" event={"ID":"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb","Type":"ContainerStarted","Data":"89f1428f44a0348296f3428ef41ec380796f177be93f31c4d5bb313f1b52a095"}
Jul 29 10:30:32.395590 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:32.395558    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kbgr9"
Jul 29 10:30:32.972962 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:32.972928    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:33.075936 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:33.075886    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:33.558377 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:33.558298    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd" event={"ID":"f553caa1-3de3-4749-aba5-eed1765f3819","Type":"ContainerStarted","Data":"1521da1f50acb035ea89c6edc87f9b4aa1b1ad5172c3158681843f078393890e"}
Jul 29 10:30:33.561383 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:33.561354    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s" event={"ID":"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb","Type":"ContainerStarted","Data":"aa212073196b14ac05de875268f8a3d9a7c202626e32ffb519a7fcafbc9457e9"}
Jul 29 10:30:33.561535 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:33.561423    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s" event={"ID":"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb","Type":"ContainerStarted","Data":"58b684ade090956059302b9b0a2b72acf73098d04b55f2857c10d9d53f49825a"}
Jul 29 10:30:33.561535 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:33.561496    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:33.561535 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:33.561533    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s" event={"ID":"9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb","Type":"ContainerStarted","Data":"63b927326e34156e6b86ddd6a7725da325857a7a6ae05437e29f0e5583922916"}
Jul 29 10:30:33.579215 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:33.579164    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd" podStartSLOduration=1.9964246270000001 podStartE2EDuration="4.579149642s" totalImagesPullingTime="2.582725015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:33.578508291 +0000 UTC m=+107.971721161" watchObservedRunningTime="2026-07-29 10:30:33.579149642 +0000 UTC m=+107.972362511"
Jul 29 10:30:33.601815 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:33.601764    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s" podStartSLOduration=1.655142493 podStartE2EDuration="5.601745766s" totalImagesPullingTime="3.946603273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:33.600690783 +0000 UTC m=+107.993903652" watchObservedRunningTime="2026-07-29 10:30:33.601745766 +0000 UTC m=+107.994958637"
Jul 29 10:30:34.499699 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:34.499662    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:34.600937 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:34.600904    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:35.624795 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:35.624759    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:35.682539 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:35.682511    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:38.126986 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:38.126952    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:38.191956 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:38.191922    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:39.570810 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:39.570780    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-bbf8fcf7d-wk55s"
Jul 29 10:30:41.021368 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:41.021330    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f96db74d-6bljz"]
Jul 29 10:30:41.161129 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:41.161090    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:41.226099 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:41.226069    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:41.279095 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:41.279013    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:49.025952 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:49.025915    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:49.767207 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:49.767172    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:49.767382 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:49.767258    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:30:50.623869 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:50.623833    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:50.725592 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:50.725561    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:56.148405 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:56.148370    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:56.212588 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:56.212555    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:57.281806 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:57.281771    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:57.382874 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:57.382841    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:30:57.439307 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:30:57.439270    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:05.628337 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:05.628302    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:05.750376 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:05.750336    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:06.046217 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.046125    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67f96db74d-6bljz" podUID="7529d802-ffcb-430f-aa31-e934ff07c894" containerName="console" containerID="cri-o://5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6" gracePeriod=15
Jul 29 10:31:06.297401 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.297340    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:31:06.458944 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.458907    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-serving-cert\") pod \"7529d802-ffcb-430f-aa31-e934ff07c894\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") "
Jul 29 10:31:06.459123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.458968    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-oauth-serving-cert\") pod \"7529d802-ffcb-430f-aa31-e934ff07c894\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") "
Jul 29 10:31:06.459123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.459008    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-console-config\") pod \"7529d802-ffcb-430f-aa31-e934ff07c894\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") "
Jul 29 10:31:06.459123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.459025    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-oauth-config\") pod \"7529d802-ffcb-430f-aa31-e934ff07c894\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") "
Jul 29 10:31:06.459123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.459039    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-service-ca\") pod \"7529d802-ffcb-430f-aa31-e934ff07c894\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") "
Jul 29 10:31:06.459123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.459094    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdds4\" (UniqueName: \"kubernetes.io/projected/7529d802-ffcb-430f-aa31-e934ff07c894-kube-api-access-qdds4\") pod \"7529d802-ffcb-430f-aa31-e934ff07c894\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") "
Jul 29 10:31:06.459389 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.459130    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-trusted-ca-bundle\") pod \"7529d802-ffcb-430f-aa31-e934ff07c894\" (UID: \"7529d802-ffcb-430f-aa31-e934ff07c894\") "
Jul 29 10:31:06.459524 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.459486    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-console-config" (OuterVolumeSpecName: "console-config") pod "7529d802-ffcb-430f-aa31-e934ff07c894" (UID: "7529d802-ffcb-430f-aa31-e934ff07c894"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:06.459650 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.459596    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7529d802-ffcb-430f-aa31-e934ff07c894" (UID: "7529d802-ffcb-430f-aa31-e934ff07c894"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:06.459742 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.459692    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-service-ca" (OuterVolumeSpecName: "service-ca") pod "7529d802-ffcb-430f-aa31-e934ff07c894" (UID: "7529d802-ffcb-430f-aa31-e934ff07c894"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:06.459742 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.459703    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7529d802-ffcb-430f-aa31-e934ff07c894" (UID: "7529d802-ffcb-430f-aa31-e934ff07c894"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:31:06.461395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.461355    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7529d802-ffcb-430f-aa31-e934ff07c894" (UID: "7529d802-ffcb-430f-aa31-e934ff07c894"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:31:06.461528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.461443    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7529d802-ffcb-430f-aa31-e934ff07c894" (UID: "7529d802-ffcb-430f-aa31-e934ff07c894"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:31:06.461698 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.461673    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7529d802-ffcb-430f-aa31-e934ff07c894-kube-api-access-qdds4" (OuterVolumeSpecName: "kube-api-access-qdds4") pod "7529d802-ffcb-430f-aa31-e934ff07c894" (UID: "7529d802-ffcb-430f-aa31-e934ff07c894"). InnerVolumeSpecName "kube-api-access-qdds4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:31:06.560658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.560576    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-trusted-ca-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:31:06.560658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.560605    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-serving-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:31:06.560658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.560616    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-oauth-serving-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:31:06.560658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.560626    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-console-config\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:31:06.560658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.560634    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7529d802-ffcb-430f-aa31-e934ff07c894-console-oauth-config\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:31:06.560658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.560642    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7529d802-ffcb-430f-aa31-e934ff07c894-service-ca\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:31:06.560658 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.560650    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdds4\" (UniqueName: \"kubernetes.io/projected/7529d802-ffcb-430f-aa31-e934ff07c894-kube-api-access-qdds4\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:31:06.694661 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.694626    2569 generic.go:356] "Generic (PLEG): container finished" podID="c2c7d066-1ffc-460b-88c0-709d3f265242" containerID="eb5861ed306fa4d776b6c981d1ac99d8232990cf5abe7c33621b23864b87d290" exitCode=0
Jul 29 10:31:06.695102 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.694702    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-5dgxd" event={"ID":"c2c7d066-1ffc-460b-88c0-709d3f265242","Type":"ContainerDied","Data":"eb5861ed306fa4d776b6c981d1ac99d8232990cf5abe7c33621b23864b87d290"}
Jul 29 10:31:06.695102 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.695087    2569 scope.go:117] "RemoveContainer" containerID="eb5861ed306fa4d776b6c981d1ac99d8232990cf5abe7c33621b23864b87d290"
Jul 29 10:31:06.696070 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.696044    2569 generic.go:356] "Generic (PLEG): container finished" podID="7529d802-ffcb-430f-aa31-e934ff07c894" containerID="5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6" exitCode=0
Jul 29 10:31:06.696200 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.696081    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f96db74d-6bljz"
Jul 29 10:31:06.696200 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.696126    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f96db74d-6bljz" event={"ID":"7529d802-ffcb-430f-aa31-e934ff07c894","Type":"ContainerDied","Data":"5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6"}
Jul 29 10:31:06.696200 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.696182    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f96db74d-6bljz" event={"ID":"7529d802-ffcb-430f-aa31-e934ff07c894","Type":"ContainerDied","Data":"d718877f3d62f984a1f9c13f7f9b325d5bd1af99eb25af56ee8d32c403a36274"}
Jul 29 10:31:06.696344 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.696207    2569 scope.go:117] "RemoveContainer" containerID="5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6"
Jul 29 10:31:06.705624 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.705603    2569 scope.go:117] "RemoveContainer" containerID="5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6"
Jul 29 10:31:06.705918 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:31:06.705879    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6\": container with ID starting with 5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6 not found: ID does not exist" containerID="5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6"
Jul 29 10:31:06.705986 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.705925    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6"} err="failed to get container status \"5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6\": rpc error: code = NotFound desc = could not find container \"5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6\": container with ID starting with 5bee608f763eb17b96d086c2f4be8641cebfd1273f6eb44a14f51724dfe5c6b6 not found: ID does not exist"
Jul 29 10:31:06.728181 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.728155    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f96db74d-6bljz"]
Jul 29 10:31:06.736178 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:06.736146    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f96db74d-6bljz"]
Jul 29 10:31:07.701604 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:07.701556    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-5dgxd" event={"ID":"c2c7d066-1ffc-460b-88c0-709d3f265242","Type":"ContainerStarted","Data":"311da0b0fb9601939e464a03aa7652a47d6371e6346b2d7b5143b44f440344b6"}
Jul 29 10:31:08.040587 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:08.040511    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7529d802-ffcb-430f-aa31-e934ff07c894" path="/var/lib/kubelet/pods/7529d802-ffcb-430f-aa31-e934ff07c894/volumes"
Jul 29 10:31:09.771857 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:09.771826    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:31:09.775684 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:09.775665    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7fb54f9c7d-2hmrd"
Jul 29 10:31:44.291663 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:44.291597    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:44.977411 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:44.977361    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:54.062653 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:54.062618    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:54.120792 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:54.120759    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:59.531545 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:59.531506    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:31:59.623642 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:31:59.623606    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:10.486205 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:32:10.486164    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:10.585409 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:32:10.585371    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:10.686169 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:32:10.686134    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:11.414689 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:32:11.414652    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:11.516048 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:32:11.516013    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:20.633827 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:32:20.633793    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:20.733604 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:32:20.733557    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:37.705953 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:32:37.705916    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:32:37.771649 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:32:37.771616    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:23.012935 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:33:23.012849    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:27.367329 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:33:27.367290    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:27.464139 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:33:27.464086    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:27.734824 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:33:27.734729    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:27.837204 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:33:27.837171    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:27.936128 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:33:27.936096    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:45.924053 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:33:45.924024    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 10:33:49.808475 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:33:49.808444    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:33:49.908369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:33:49.908323    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:05.078906 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:05.078848    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:06.195479 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:06.195420    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:06.294894 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:06.294845    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:06.423202 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:06.423169    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:06.521391 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:06.521288    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:06.811669 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:06.811574    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:07.068058 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:07.067958    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:07.168038 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:07.168003    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:07.269077 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:07.269046    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:18.371150 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:18.371112    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:18.470759 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:18.470722    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:57.914975 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:57.914937    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:34:59.019840 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:34:59.019800    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:23.386200 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.386119    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:23.455158 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.455120    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-65f566c74b-v6w4s"]
Jul 29 10:35:23.455588 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.455569    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7529d802-ffcb-430f-aa31-e934ff07c894" containerName="console"
Jul 29 10:35:23.455750 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.455733    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7529d802-ffcb-430f-aa31-e934ff07c894" containerName="console"
Jul 29 10:35:23.455837 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.455817    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7529d802-ffcb-430f-aa31-e934ff07c894" containerName="console"
Jul 29 10:35:23.458833 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.458814    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:35:23.461784 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.461761    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 10:35:23.461912 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.461896    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 10:35:23.462951 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.462932    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-pp2rd\""
Jul 29 10:35:23.463041 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.462959    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 29 10:35:23.467546 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.467524    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-65f566c74b-v6w4s"]
Jul 29 10:35:23.469025 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.468998    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:23.492525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.492499    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-r6dnp"]
Jul 29 10:35:23.495609 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.495590    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:23.498310 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.498285    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 29 10:35:23.498420 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.498344    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dqz2x\""
Jul 29 10:35:23.506123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.506097    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-r6dnp"]
Jul 29 10:35:23.507735 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.507713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqgkw\" (UniqueName: \"kubernetes.io/projected/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-kube-api-access-wqgkw\") pod \"kserve-controller-manager-65f566c74b-v6w4s\" (UID: \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\") " pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:35:23.507830 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.507744    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-cert\") pod \"kserve-controller-manager-65f566c74b-v6w4s\" (UID: \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\") " pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:35:23.551471 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.551441    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:23.608130 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.608099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/170bf724-f8cc-45e8-9649-bf48faeef1ba-data\") pod \"seaweedfs-64568bcd49-r6dnp\" (UID: \"170bf724-f8cc-45e8-9649-bf48faeef1ba\") " pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:23.608355 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.608172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqgkw\" (UniqueName: \"kubernetes.io/projected/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-kube-api-access-wqgkw\") pod \"kserve-controller-manager-65f566c74b-v6w4s\" (UID: \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\") " pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:35:23.608355 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.608197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgrt\" (UniqueName: \"kubernetes.io/projected/170bf724-f8cc-45e8-9649-bf48faeef1ba-kube-api-access-bmgrt\") pod \"seaweedfs-64568bcd49-r6dnp\" (UID: \"170bf724-f8cc-45e8-9649-bf48faeef1ba\") " pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:23.608355 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.608225    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-cert\") pod \"kserve-controller-manager-65f566c74b-v6w4s\" (UID: \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\") " pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:35:23.608515 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.608449    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:23.610721 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.610692    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-cert\") pod \"kserve-controller-manager-65f566c74b-v6w4s\" (UID: \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\") " pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:35:23.617092 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.617069    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqgkw\" (UniqueName: \"kubernetes.io/projected/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-kube-api-access-wqgkw\") pod \"kserve-controller-manager-65f566c74b-v6w4s\" (UID: \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\") " pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:35:23.709391 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.709298    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/170bf724-f8cc-45e8-9649-bf48faeef1ba-data\") pod \"seaweedfs-64568bcd49-r6dnp\" (UID: \"170bf724-f8cc-45e8-9649-bf48faeef1ba\") " pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:23.709391 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.709375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgrt\" (UniqueName: \"kubernetes.io/projected/170bf724-f8cc-45e8-9649-bf48faeef1ba-kube-api-access-bmgrt\") pod \"seaweedfs-64568bcd49-r6dnp\" (UID: \"170bf724-f8cc-45e8-9649-bf48faeef1ba\") " pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:23.709716 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.709696    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/170bf724-f8cc-45e8-9649-bf48faeef1ba-data\") pod \"seaweedfs-64568bcd49-r6dnp\" (UID: \"170bf724-f8cc-45e8-9649-bf48faeef1ba\") " pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:23.721045 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.721013    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgrt\" (UniqueName: \"kubernetes.io/projected/170bf724-f8cc-45e8-9649-bf48faeef1ba-kube-api-access-bmgrt\") pod \"seaweedfs-64568bcd49-r6dnp\" (UID: \"170bf724-f8cc-45e8-9649-bf48faeef1ba\") " pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:23.770916 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.770882    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:35:23.805128 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.805092    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:23.919679 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.919547    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-65f566c74b-v6w4s"]
Jul 29 10:35:23.921998 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:35:23.921965    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9611c2_d4b3_4945_8b28_eb1af6cbaa6d.slice/crio-1566633c096c3e331b532d332d4508e4102556987920f449a5e7869a647e95d3 WatchSource:0}: Error finding container 1566633c096c3e331b532d332d4508e4102556987920f449a5e7869a647e95d3: Status 404 returned error can't find the container with id 1566633c096c3e331b532d332d4508e4102556987920f449a5e7869a647e95d3
Jul 29 10:35:23.923886 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.923860    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:35:23.950436 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:23.950401    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-r6dnp"]
Jul 29 10:35:23.952706 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:35:23.952683    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170bf724_f8cc_45e8_9649_bf48faeef1ba.slice/crio-b79f36509b339a9793253fd858b06f12190b750c9793ea96f2ee4ecd5cbca754 WatchSource:0}: Error finding container b79f36509b339a9793253fd858b06f12190b750c9793ea96f2ee4ecd5cbca754: Status 404 returned error can't find the container with id b79f36509b339a9793253fd858b06f12190b750c9793ea96f2ee4ecd5cbca754
Jul 29 10:35:24.292790 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:24.292756    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:24.394068 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:24.394039    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:24.451069 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:24.451032    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:24.663826 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:24.663778    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-r6dnp" event={"ID":"170bf724-f8cc-45e8-9649-bf48faeef1ba","Type":"ContainerStarted","Data":"b79f36509b339a9793253fd858b06f12190b750c9793ea96f2ee4ecd5cbca754"}
Jul 29 10:35:24.668118 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:24.667974    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s" event={"ID":"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d","Type":"ContainerStarted","Data":"1566633c096c3e331b532d332d4508e4102556987920f449a5e7869a647e95d3"}
Jul 29 10:35:27.682069 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:27.682028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s" event={"ID":"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d","Type":"ContainerStarted","Data":"596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056"}
Jul 29 10:35:27.682408 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:27.682167    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:35:27.699682 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:27.699565    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s" podStartSLOduration=1.071774632 podStartE2EDuration="4.699544628s" totalImagesPullingTime="3.627769996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:35:27.699339248 +0000 UTC m=+402.092552118" watchObservedRunningTime="2026-07-29 10:35:27.699544628 +0000 UTC m=+402.092757504"
Jul 29 10:35:28.686797 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:28.686760    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-r6dnp" event={"ID":"170bf724-f8cc-45e8-9649-bf48faeef1ba","Type":"ContainerStarted","Data":"5c822ae296d7ea1024691fce7b48459f30ae1b4f441c48672ce39fd3f4313d76"}
Jul 29 10:35:28.687262 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:28.686946    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:28.703127 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:28.703079    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-r6dnp" podStartSLOduration=2.014459982 podStartE2EDuration="5.703063247s" totalImagesPullingTime="3.688603265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:35:28.702406846 +0000 UTC m=+403.095619716" watchObservedRunningTime="2026-07-29 10:35:28.703063247 +0000 UTC m=+403.096276117"
Jul 29 10:35:34.693038 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:34.693002    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-r6dnp"
Jul 29 10:35:46.209674 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:46.209640    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:46.311577 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:46.311540    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:35:58.692047 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:35:58.692017    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:36:07.294734 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:07.294696    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:07.397811 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:07.397779    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:18.458256 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:18.458219    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:19.345998 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:19.345965    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:20.402874 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.402843    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"]
Jul 29 10:36:20.405689 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.405673    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"
Jul 29 10:36:20.409681 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.409657    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pb4dl\""
Jul 29 10:36:20.409681 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.409661    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:36:20.409953 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.409938    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 29 10:36:20.418978 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.418958    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"]
Jul 29 10:36:20.474694 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.474662    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9201e11a-6be6-4ce8-ac42-ce23e186425e-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-z6v4p\" (UID: \"9201e11a-6be6-4ce8-ac42-ce23e186425e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"
Jul 29 10:36:20.474694 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.474695    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwpbf\" (UniqueName: \"kubernetes.io/projected/9201e11a-6be6-4ce8-ac42-ce23e186425e-kube-api-access-dwpbf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-z6v4p\" (UID: \"9201e11a-6be6-4ce8-ac42-ce23e186425e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"
Jul 29 10:36:20.575123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.575087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9201e11a-6be6-4ce8-ac42-ce23e186425e-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-z6v4p\" (UID: \"9201e11a-6be6-4ce8-ac42-ce23e186425e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"
Jul 29 10:36:20.575123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.575124    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwpbf\" (UniqueName: \"kubernetes.io/projected/9201e11a-6be6-4ce8-ac42-ce23e186425e-kube-api-access-dwpbf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-z6v4p\" (UID: \"9201e11a-6be6-4ce8-ac42-ce23e186425e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"
Jul 29 10:36:20.575513 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.575482    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9201e11a-6be6-4ce8-ac42-ce23e186425e-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-z6v4p\" (UID: \"9201e11a-6be6-4ce8-ac42-ce23e186425e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"
Jul 29 10:36:20.585480 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.585453    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwpbf\" (UniqueName: \"kubernetes.io/projected/9201e11a-6be6-4ce8-ac42-ce23e186425e-kube-api-access-dwpbf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-z6v4p\" (UID: \"9201e11a-6be6-4ce8-ac42-ce23e186425e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"
Jul 29 10:36:20.715525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.715443    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"
Jul 29 10:36:20.848629 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.848599    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p"]
Jul 29 10:36:20.852488 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:36:20.852456    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9201e11a_6be6_4ce8_ac42_ce23e186425e.slice/crio-96d3a08ede4c3b2db0e794209b9f8aa1094e877a8d47f27feb9647d458ecd2eb WatchSource:0}: Error finding container 96d3a08ede4c3b2db0e794209b9f8aa1094e877a8d47f27feb9647d458ecd2eb: Status 404 returned error can't find the container with id 96d3a08ede4c3b2db0e794209b9f8aa1094e877a8d47f27feb9647d458ecd2eb
Jul 29 10:36:20.877397 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:20.877367    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p" event={"ID":"9201e11a-6be6-4ce8-ac42-ce23e186425e","Type":"ContainerStarted","Data":"96d3a08ede4c3b2db0e794209b9f8aa1094e877a8d47f27feb9647d458ecd2eb"}
Jul 29 10:36:22.185179 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:22.185116    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:22.292055 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:22.292023    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:23.894941 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:23.894903    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p" event={"ID":"9201e11a-6be6-4ce8-ac42-ce23e186425e","Type":"ContainerStarted","Data":"c86f0dd93e54c7732ce1144df65ed6462c73cb7e5ba22e1848df693b4cbbdf5d"}
Jul 29 10:36:23.918055 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:23.917994    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-z6v4p" podStartSLOduration=1.335543259 podStartE2EDuration="3.917973565s" totalImagesPullingTime="2.582430306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:36:23.915987068 +0000 UTC m=+458.309199947" watchObservedRunningTime="2026-07-29 10:36:23.917973565 +0000 UTC m=+458.311186440"
Jul 29 10:36:24.401097 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:24.401063    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:24.530313 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:24.530277    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:26.646000 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:26.645963    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:26.932597 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:26.932501    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:27.035525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:27.035485    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:27.133511 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:27.133475    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:28.783343 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:28.783312    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:28.890390 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:28.890358    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:28.996439 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:28.996404    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:29.621666 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.621631    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"]
Jul 29 10:36:29.624949 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.624932    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"
Jul 29 10:36:29.627720 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.627700    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 10:36:29.628733 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.628711    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bf5lx\""
Jul 29 10:36:29.628832 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.628750    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 10:36:29.632755 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.632735    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"]
Jul 29 10:36:29.654588 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.654559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8d25l\" (UID: \"ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"
Jul 29 10:36:29.654761 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.654612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gnpl\" (UniqueName: \"kubernetes.io/projected/ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff-kube-api-access-7gnpl\") pod \"cert-manager-cainjector-559cdd8c87-8d25l\" (UID: \"ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"
Jul 29 10:36:29.755782 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.755745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gnpl\" (UniqueName: \"kubernetes.io/projected/ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff-kube-api-access-7gnpl\") pod \"cert-manager-cainjector-559cdd8c87-8d25l\" (UID: \"ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"
Jul 29 10:36:29.755952 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.755821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8d25l\" (UID: \"ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"
Jul 29 10:36:29.765307 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.765267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8d25l\" (UID: \"ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"
Jul 29 10:36:29.770978 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.770945    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gnpl\" (UniqueName: \"kubernetes.io/projected/ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff-kube-api-access-7gnpl\") pod \"cert-manager-cainjector-559cdd8c87-8d25l\" (UID: \"ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"
Jul 29 10:36:29.950485 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:29.950369    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"
Jul 29 10:36:30.088539 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:30.088502    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8d25l"]
Jul 29 10:36:30.093050 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:36:30.093018    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff5e934c_182b_4c9e_a7ea_c5df9c1c87ff.slice/crio-8505826cbf1e80d5d51e0a0dad41b6d93d72689576b0253710da08bebff93d77 WatchSource:0}: Error finding container 8505826cbf1e80d5d51e0a0dad41b6d93d72689576b0253710da08bebff93d77: Status 404 returned error can't find the container with id 8505826cbf1e80d5d51e0a0dad41b6d93d72689576b0253710da08bebff93d77
Jul 29 10:36:30.926523 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:30.926478    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l" event={"ID":"ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff","Type":"ContainerStarted","Data":"8505826cbf1e80d5d51e0a0dad41b6d93d72689576b0253710da08bebff93d77"}
Jul 29 10:36:32.937079 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:32.937030    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l" event={"ID":"ff5e934c-182b-4c9e-a7ea-c5df9c1c87ff","Type":"ContainerStarted","Data":"c4681089f1547192eaa05d6536e741f9ae54e5d6872ea923593f65035ae589bf"}
Jul 29 10:36:32.954671 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:32.954613    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8d25l" podStartSLOduration=1.316025826 podStartE2EDuration="3.954593474s" totalImagesPullingTime="2.638567648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:36:32.953339558 +0000 UTC m=+467.346552453" watchObservedRunningTime="2026-07-29 10:36:32.954593474 +0000 UTC m=+467.347806344"
Jul 29 10:36:34.145041 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:34.145008    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:35.000047 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.000004    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w24ns"]
Jul 29 10:36:35.003558 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.003530    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w24ns"
Jul 29 10:36:35.006081 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.006057    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wdppj\""
Jul 29 10:36:35.012993 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.012969    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w24ns"]
Jul 29 10:36:35.103249 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.103209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e53802d-4107-431c-8653-b1d1fb57610b-bound-sa-token\") pod \"cert-manager-55b4d954d5-w24ns\" (UID: \"9e53802d-4107-431c-8653-b1d1fb57610b\") " pod="cert-manager/cert-manager-55b4d954d5-w24ns"
Jul 29 10:36:35.103421 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.103272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nnz4\" (UniqueName: \"kubernetes.io/projected/9e53802d-4107-431c-8653-b1d1fb57610b-kube-api-access-5nnz4\") pod \"cert-manager-55b4d954d5-w24ns\" (UID: \"9e53802d-4107-431c-8653-b1d1fb57610b\") " pod="cert-manager/cert-manager-55b4d954d5-w24ns"
Jul 29 10:36:35.204571 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.204529    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nnz4\" (UniqueName: \"kubernetes.io/projected/9e53802d-4107-431c-8653-b1d1fb57610b-kube-api-access-5nnz4\") pod \"cert-manager-55b4d954d5-w24ns\" (UID: \"9e53802d-4107-431c-8653-b1d1fb57610b\") " pod="cert-manager/cert-manager-55b4d954d5-w24ns"
Jul 29 10:36:35.204956 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.204618    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e53802d-4107-431c-8653-b1d1fb57610b-bound-sa-token\") pod \"cert-manager-55b4d954d5-w24ns\" (UID: \"9e53802d-4107-431c-8653-b1d1fb57610b\") " pod="cert-manager/cert-manager-55b4d954d5-w24ns"
Jul 29 10:36:35.213144 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.213115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e53802d-4107-431c-8653-b1d1fb57610b-bound-sa-token\") pod \"cert-manager-55b4d954d5-w24ns\" (UID: \"9e53802d-4107-431c-8653-b1d1fb57610b\") " pod="cert-manager/cert-manager-55b4d954d5-w24ns"
Jul 29 10:36:35.213306 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.213241    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nnz4\" (UniqueName: \"kubernetes.io/projected/9e53802d-4107-431c-8653-b1d1fb57610b-kube-api-access-5nnz4\") pod \"cert-manager-55b4d954d5-w24ns\" (UID: \"9e53802d-4107-431c-8653-b1d1fb57610b\") " pod="cert-manager/cert-manager-55b4d954d5-w24ns"
Jul 29 10:36:35.313414 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.313317    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w24ns"
Jul 29 10:36:35.446839 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.446813    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w24ns"]
Jul 29 10:36:35.449281 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:36:35.449250    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e53802d_4107_431c_8653_b1d1fb57610b.slice/crio-1e9115a4a73081978c528fce81d45a68a02ad3db8b4bb873e267a7806978a7f5 WatchSource:0}: Error finding container 1e9115a4a73081978c528fce81d45a68a02ad3db8b4bb873e267a7806978a7f5: Status 404 returned error can't find the container with id 1e9115a4a73081978c528fce81d45a68a02ad3db8b4bb873e267a7806978a7f5
Jul 29 10:36:35.952672 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.952635    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w24ns" event={"ID":"9e53802d-4107-431c-8653-b1d1fb57610b","Type":"ContainerStarted","Data":"b001a990212883171eb8a070fefcdd23a8793d5a1975e677c22e2ecdd2ee8d70"}
Jul 29 10:36:35.952672 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.952676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w24ns" event={"ID":"9e53802d-4107-431c-8653-b1d1fb57610b","Type":"ContainerStarted","Data":"1e9115a4a73081978c528fce81d45a68a02ad3db8b4bb873e267a7806978a7f5"}
Jul 29 10:36:35.970525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:35.970463    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-w24ns" podStartSLOduration=1.9704053620000002 podStartE2EDuration="1.970405362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:36:35.969329699 +0000 UTC m=+470.362542584" watchObservedRunningTime="2026-07-29 10:36:35.970405362 +0000 UTC m=+470.363618232"
Jul 29 10:36:37.915281 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:37.915248    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:37.973247 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:37.973214    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:40.662690 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:40.662649    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:40.761423 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:40.761382    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:42.878685 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:42.878650    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"]
Jul 29 10:36:42.881909 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:42.881892    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"
Jul 29 10:36:42.885959 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:42.885935    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kbrww\""
Jul 29 10:36:42.885959 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:42.885935    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:36:42.887222 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:42.887200    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 10:36:42.889779 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:42.889753    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"]
Jul 29 10:36:42.972532 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:42.972499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad3fa79-f971-4c2f-9aee-4e7199326813-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rd9zg\" (UID: \"5ad3fa79-f971-4c2f-9aee-4e7199326813\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"
Jul 29 10:36:42.972715 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:42.972565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpx2\" (UniqueName: \"kubernetes.io/projected/5ad3fa79-f971-4c2f-9aee-4e7199326813-kube-api-access-8jpx2\") pod \"openshift-lws-operator-fd8ccff4c-rd9zg\" (UID: \"5ad3fa79-f971-4c2f-9aee-4e7199326813\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"
Jul 29 10:36:43.073298 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:43.073260    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad3fa79-f971-4c2f-9aee-4e7199326813-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rd9zg\" (UID: \"5ad3fa79-f971-4c2f-9aee-4e7199326813\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"
Jul 29 10:36:43.073507 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:43.073447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpx2\" (UniqueName: \"kubernetes.io/projected/5ad3fa79-f971-4c2f-9aee-4e7199326813-kube-api-access-8jpx2\") pod \"openshift-lws-operator-fd8ccff4c-rd9zg\" (UID: \"5ad3fa79-f971-4c2f-9aee-4e7199326813\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"
Jul 29 10:36:43.073690 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:43.073670    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad3fa79-f971-4c2f-9aee-4e7199326813-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rd9zg\" (UID: \"5ad3fa79-f971-4c2f-9aee-4e7199326813\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"
Jul 29 10:36:43.082801 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:43.082777    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpx2\" (UniqueName: \"kubernetes.io/projected/5ad3fa79-f971-4c2f-9aee-4e7199326813-kube-api-access-8jpx2\") pod \"openshift-lws-operator-fd8ccff4c-rd9zg\" (UID: \"5ad3fa79-f971-4c2f-9aee-4e7199326813\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"
Jul 29 10:36:43.192466 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:43.192349    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"
Jul 29 10:36:43.321700 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:43.321677    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg"]
Jul 29 10:36:43.324377 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:36:43.324351    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ad3fa79_f971_4c2f_9aee_4e7199326813.slice/crio-f4a42565f8218f8247c41974c0899a515a26eb383e27de4977f9447bdcc8cd3b WatchSource:0}: Error finding container f4a42565f8218f8247c41974c0899a515a26eb383e27de4977f9447bdcc8cd3b: Status 404 returned error can't find the container with id f4a42565f8218f8247c41974c0899a515a26eb383e27de4977f9447bdcc8cd3b
Jul 29 10:36:43.989933 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:43.989901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg" event={"ID":"5ad3fa79-f971-4c2f-9aee-4e7199326813","Type":"ContainerStarted","Data":"f4a42565f8218f8247c41974c0899a515a26eb383e27de4977f9447bdcc8cd3b"}
Jul 29 10:36:47.004848 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:47.004806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg" event={"ID":"5ad3fa79-f971-4c2f-9aee-4e7199326813","Type":"ContainerStarted","Data":"69fc81c0fcf8ed80ae7536b40e790578cffbff75b849cca91eb6c98d39060863"}
Jul 29 10:36:47.025076 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:47.025023    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rd9zg" podStartSLOduration=1.457765143 podStartE2EDuration="5.025007039s" totalImagesPullingTime="3.567241896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:36:47.023579883 +0000 UTC m=+481.416792794" watchObservedRunningTime="2026-07-29 10:36:47.025007039 +0000 UTC m=+481.418219964"
Jul 29 10:36:52.342327 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:52.342290    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:56.565329 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:56.565291    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:56.653680 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:56.653616    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:57.667769 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.667734    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"]
Jul 29 10:36:57.670443 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.670403    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:57.674101 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.674083    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.677143 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.677121    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 29 10:36:57.677637 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.677616    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q69px\""
Jul 29 10:36:57.677763 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.677665    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 29 10:36:57.677841 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.677662    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 29 10:36:57.678279 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.678086    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 10:36:57.689988 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.689964    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"]
Jul 29 10:36:57.763471 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.763404    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:57.799472 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.799421    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a1895cc-3ece-4c53-9be4-784aa44da7ca-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.799585 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.799480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqsq7\" (UniqueName: \"kubernetes.io/projected/3a1895cc-3ece-4c53-9be4-784aa44da7ca-kube-api-access-vqsq7\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.799585 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.799518    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.799585 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.799568    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.799703 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.799593    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1895cc-3ece-4c53-9be4-784aa44da7ca-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.799703 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.799634    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.799703 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.799664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.900226 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.900182    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1895cc-3ece-4c53-9be4-784aa44da7ca-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.900226 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.900226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.900503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.900246    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.900503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.900306    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a1895cc-3ece-4c53-9be4-784aa44da7ca-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.900503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.900322    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqsq7\" (UniqueName: \"kubernetes.io/projected/3a1895cc-3ece-4c53-9be4-784aa44da7ca-kube-api-access-vqsq7\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.900503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.900343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.900503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.900381    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.901062 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.901033    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.902698 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.902670    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1895cc-3ece-4c53-9be4-784aa44da7ca-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.902815 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.902775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.902879 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.902839    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a1895cc-3ece-4c53-9be4-784aa44da7ca-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.902879 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.902855    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.912615 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.912588    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1895cc-3ece-4c53-9be4-784aa44da7ca-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.913086 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.913065    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqsq7\" (UniqueName: \"kubernetes.io/projected/3a1895cc-3ece-4c53-9be4-784aa44da7ca-kube-api-access-vqsq7\") pod \"istiod-openshift-gateway-5677ccb49b-46d7p\" (UID: \"3a1895cc-3ece-4c53-9be4-784aa44da7ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:57.984078 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:57.983988    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:36:58.124907 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:58.124885    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"]
Jul 29 10:36:58.127351 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:36:58.127322    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1895cc_3ece_4c53_9be4_784aa44da7ca.slice/crio-354abdf54b9bcdbb16320dc740cbec6396e100b721a43fe61997cedb1e808a51 WatchSource:0}: Error finding container 354abdf54b9bcdbb16320dc740cbec6396e100b721a43fe61997cedb1e808a51: Status 404 returned error can't find the container with id 354abdf54b9bcdbb16320dc740cbec6396e100b721a43fe61997cedb1e808a51
Jul 29 10:36:58.892119 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:58.892089    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:36:59.061335 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:36:59.061302    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p" event={"ID":"3a1895cc-3ece-4c53-9be4-784aa44da7ca","Type":"ContainerStarted","Data":"354abdf54b9bcdbb16320dc740cbec6396e100b721a43fe61997cedb1e808a51"}
Jul 29 10:37:00.906998 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:00.906958    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:37:00.907264 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:00.907030    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:37:01.072843 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:01.072813    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p" event={"ID":"3a1895cc-3ece-4c53-9be4-784aa44da7ca","Type":"ContainerStarted","Data":"e23902d1843cc0ef807222bd7507312962fa52d7b83903ded47fa0713310808c"}
Jul 29 10:37:01.073040 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:01.073021    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:37:01.074528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:01.074498    2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-46d7p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 10:37:01.074660 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:01.074554    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p" podUID="3a1895cc-3ece-4c53-9be4-784aa44da7ca" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:37:01.094931 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:01.094886    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p" podStartSLOduration=1.317366096 podStartE2EDuration="4.094871236s" totalImagesPullingTime="2.77750514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:01.094784 +0000 UTC m=+495.487996890" watchObservedRunningTime="2026-07-29 10:37:01.094871236 +0000 UTC m=+495.488084097"
Jul 29 10:37:01.328772 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:01.328694    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:01.428547 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:01.428513    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:02.077674 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:02.077643    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46d7p"
Jul 29 10:37:03.803850 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:03.803819    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:03.918662 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:03.918626    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:06.916725 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:06.916696    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:09.274185 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:09.274148    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:09.374506 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:09.374460    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:24.322546 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:24.322507    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:31.108857 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:31.108823    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:33.113296 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.113259    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"]
Jul 29 10:37:33.121526 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.121502    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"
Jul 29 10:37:33.124383 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.124357    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 10:37:33.124552 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.124384    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 10:37:33.124552 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.124358    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 29 10:37:33.125599 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.125578    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-z7fjm\""
Jul 29 10:37:33.128991 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.128955    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"]
Jul 29 10:37:33.196208 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.196170    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24tq\" (UniqueName: \"kubernetes.io/projected/1dedce6f-c591-4285-b6a4-b88862969921-kube-api-access-v24tq\") pod \"dns-operator-controller-manager-ccbdf686-9wzdw\" (UID: \"1dedce6f-c591-4285-b6a4-b88862969921\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"
Jul 29 10:37:33.296952 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.296919    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24tq\" (UniqueName: \"kubernetes.io/projected/1dedce6f-c591-4285-b6a4-b88862969921-kube-api-access-v24tq\") pod \"dns-operator-controller-manager-ccbdf686-9wzdw\" (UID: \"1dedce6f-c591-4285-b6a4-b88862969921\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"
Jul 29 10:37:33.309568 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.309532    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24tq\" (UniqueName: \"kubernetes.io/projected/1dedce6f-c591-4285-b6a4-b88862969921-kube-api-access-v24tq\") pod \"dns-operator-controller-manager-ccbdf686-9wzdw\" (UID: \"1dedce6f-c591-4285-b6a4-b88862969921\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"
Jul 29 10:37:33.432503 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.432375    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"
Jul 29 10:37:33.588973 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.588945    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"]
Jul 29 10:37:33.591447 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:37:33.591404    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dedce6f_c591_4285_b6a4_b88862969921.slice/crio-d87d62da357976853e49ca8b1cdcabf11c64cd120c67556dd5374ecfa851780d WatchSource:0}: Error finding container d87d62da357976853e49ca8b1cdcabf11c64cd120c67556dd5374ecfa851780d: Status 404 returned error can't find the container with id d87d62da357976853e49ca8b1cdcabf11c64cd120c67556dd5374ecfa851780d
Jul 29 10:37:33.669614 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:33.669548    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:34.086049 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:34.086011    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:34.219162 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:34.219123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw" event={"ID":"1dedce6f-c591-4285-b6a4-b88862969921","Type":"ContainerStarted","Data":"d87d62da357976853e49ca8b1cdcabf11c64cd120c67556dd5374ecfa851780d"}
Jul 29 10:37:34.459477 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:34.459373    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:35.374160 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:35.374110    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:37.139738 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.139681    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:37.286633 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.286590    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:37.616753 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.616718    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"]
Jul 29 10:37:37.625404 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.625377    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:37.629105 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.629077    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wfg9x\""
Jul 29 10:37:37.635942 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.635914    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"]
Jul 29 10:37:37.734380 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.734346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26qf\" (UniqueName: \"kubernetes.io/projected/0ed5e62a-0495-410f-bea8-c776f9bdee17-kube-api-access-s26qf\") pod \"kuadrant-operator-controller-manager-66866f5fcc-m5dpg\" (UID: \"0ed5e62a-0495-410f-bea8-c776f9bdee17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:37.734571 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.734452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed5e62a-0495-410f-bea8-c776f9bdee17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-m5dpg\" (UID: \"0ed5e62a-0495-410f-bea8-c776f9bdee17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:37.835603 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.835568    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed5e62a-0495-410f-bea8-c776f9bdee17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-m5dpg\" (UID: \"0ed5e62a-0495-410f-bea8-c776f9bdee17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:37.835797 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.835634    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26qf\" (UniqueName: \"kubernetes.io/projected/0ed5e62a-0495-410f-bea8-c776f9bdee17-kube-api-access-s26qf\") pod \"kuadrant-operator-controller-manager-66866f5fcc-m5dpg\" (UID: \"0ed5e62a-0495-410f-bea8-c776f9bdee17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:37.835971 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.835952    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed5e62a-0495-410f-bea8-c776f9bdee17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-m5dpg\" (UID: \"0ed5e62a-0495-410f-bea8-c776f9bdee17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:37.849399 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.849366    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26qf\" (UniqueName: \"kubernetes.io/projected/0ed5e62a-0495-410f-bea8-c776f9bdee17-kube-api-access-s26qf\") pod \"kuadrant-operator-controller-manager-66866f5fcc-m5dpg\" (UID: \"0ed5e62a-0495-410f-bea8-c776f9bdee17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:37.898710 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.898622    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:37.937504 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:37.937462    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:38.053700 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:38.053651    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:38.092193 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:38.092165    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"]
Jul 29 10:37:38.093042 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:37:38.093005    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed5e62a_0495_410f_bea8_c776f9bdee17.slice/crio-7bea1cc0169049411fb5854021f2e7c2c496eb5c9d30b0792edc0fe10045de2e WatchSource:0}: Error finding container 7bea1cc0169049411fb5854021f2e7c2c496eb5c9d30b0792edc0fe10045de2e: Status 404 returned error can't find the container with id 7bea1cc0169049411fb5854021f2e7c2c496eb5c9d30b0792edc0fe10045de2e
Jul 29 10:37:38.238091 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:38.238001    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg" event={"ID":"0ed5e62a-0495-410f-bea8-c776f9bdee17","Type":"ContainerStarted","Data":"7bea1cc0169049411fb5854021f2e7c2c496eb5c9d30b0792edc0fe10045de2e"}
Jul 29 10:37:38.697920 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:38.697884    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:40.214200 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.214162    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-rzd9v"]
Jul 29 10:37:40.221343 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.221282    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v"
Jul 29 10:37:40.225089 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.225057    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4f6z8\""
Jul 29 10:37:40.232873 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.232349    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-rzd9v"]
Jul 29 10:37:40.258419 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.258375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rkxz\" (UniqueName: \"kubernetes.io/projected/3f55bcf0-6e72-49da-b427-04f7a3419901-kube-api-access-4rkxz\") pod \"authorino-operator-56fb6f6856-rzd9v\" (UID: \"3f55bcf0-6e72-49da-b427-04f7a3419901\") " pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v"
Jul 29 10:37:40.359134 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.359068    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rkxz\" (UniqueName: \"kubernetes.io/projected/3f55bcf0-6e72-49da-b427-04f7a3419901-kube-api-access-4rkxz\") pod \"authorino-operator-56fb6f6856-rzd9v\" (UID: \"3f55bcf0-6e72-49da-b427-04f7a3419901\") " pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v"
Jul 29 10:37:40.375143 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.375075    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rkxz\" (UniqueName: \"kubernetes.io/projected/3f55bcf0-6e72-49da-b427-04f7a3419901-kube-api-access-4rkxz\") pod \"authorino-operator-56fb6f6856-rzd9v\" (UID: \"3f55bcf0-6e72-49da-b427-04f7a3419901\") " pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v"
Jul 29 10:37:40.538106 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.538012    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v"
Jul 29 10:37:40.673124 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.673045    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:40.776279 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:40.776234    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:41.056036 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:41.056012    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-rzd9v"]
Jul 29 10:37:41.059185 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:37:41.059157    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f55bcf0_6e72_49da_b427_04f7a3419901.slice/crio-77bbe2d807124c3476cce5dcf5455e83a31e0753203ff572d71d96628594e2fa WatchSource:0}: Error finding container 77bbe2d807124c3476cce5dcf5455e83a31e0753203ff572d71d96628594e2fa: Status 404 returned error can't find the container with id 77bbe2d807124c3476cce5dcf5455e83a31e0753203ff572d71d96628594e2fa
Jul 29 10:37:41.256351 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:41.256276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v" event={"ID":"3f55bcf0-6e72-49da-b427-04f7a3419901","Type":"ContainerStarted","Data":"77bbe2d807124c3476cce5dcf5455e83a31e0753203ff572d71d96628594e2fa"}
Jul 29 10:37:43.267392 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:43.267364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg" event={"ID":"0ed5e62a-0495-410f-bea8-c776f9bdee17","Type":"ContainerStarted","Data":"9f552a018b7cb287625e8eabab3db09c8c4c3247a90c6c2c727ee7721dfeeb26"}
Jul 29 10:37:43.268026 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:43.267999    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:43.288746 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:43.288697    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg" podStartSLOduration=1.820181864 podStartE2EDuration="6.288680868s" totalImagesPullingTime="4.468499004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:43.286906515 +0000 UTC m=+537.680119398" watchObservedRunningTime="2026-07-29 10:37:43.288680868 +0000 UTC m=+537.681893737"
Jul 29 10:37:43.666090 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:43.666054    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:43.771614 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:43.771583    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:44.274028 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:44.273985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v" event={"ID":"3f55bcf0-6e72-49da-b427-04f7a3419901","Type":"ContainerStarted","Data":"f444637036c451ebeb65e4c8a49b44dd6a914c5adf339be7273cadf2cbbe9ad6"}
Jul 29 10:37:44.274525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:44.274157    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v"
Jul 29 10:37:44.294679 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:44.294626    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v" podStartSLOduration=2.093295372 podStartE2EDuration="4.294610864s" totalImagesPullingTime="2.201315492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:44.292912759 +0000 UTC m=+538.686125639" watchObservedRunningTime="2026-07-29 10:37:44.294610864 +0000 UTC m=+538.687823733"
Jul 29 10:37:44.952077 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:44.952046    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:45.350992 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:45.350962    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:48.864158 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:48.864120    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:48.916857 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:48.916796    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:49.059575 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:49.059543    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:49.131651 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:49.131575    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:49.267466 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:49.267420    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:49.370346 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:49.370304    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:54.276644 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:54.276615    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-m5dpg"
Jul 29 10:37:55.280826 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:55.280797    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-rzd9v"
Jul 29 10:37:57.148148 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:57.148114    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:57.252607 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:57.252576    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:59.755366 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:59.755342    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 29 10:37:59.780370 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:59.780346    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:37:59.880659 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:37:59.880627    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:00.110528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:00.110491    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:00.216212 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:00.216177    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:00.346098 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:00.346064    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw" event={"ID":"1dedce6f-c591-4285-b6a4-b88862969921","Type":"ContainerStarted","Data":"b1abf6e9562b2715acaf57ef6447f8b0ce04dca227d3b1857f5c3498e472685f"}
Jul 29 10:38:00.346277 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:00.346171    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"
Jul 29 10:38:00.364832 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:00.364734    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw" podStartSLOduration=1.206047874 podStartE2EDuration="27.36471806s" totalImagesPullingTime="26.158670186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:00.362958973 +0000 UTC m=+554.756171844" watchObservedRunningTime="2026-07-29 10:38:00.36471806 +0000 UTC m=+554.757930930"
Jul 29 10:38:02.718494 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:02.718448    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:02.823500 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:02.823466    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:08.121123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:08.121093    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:08.222065 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:08.222033    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:08.520743 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:08.520666    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:08.620413 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:08.620387    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:08.955780 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:08.955740    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:09.054025 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:09.053992    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:11.353492 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:11.353459    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-9wzdw"
Jul 29 10:38:16.936334 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:16.936302    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:17.038130 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:17.038095    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:19.932156 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:19.932123    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:20.032937 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:20.032898    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:26.816061 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.816011    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:26.816418 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.816267    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-mks6d"]
Jul 29 10:38:26.820564 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.820546    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:26.824523 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.824501    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k4wxg\""
Jul 29 10:38:26.824684 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.824525    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 10:38:26.830108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.830086    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-mks6d"]
Jul 29 10:38:26.872126 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.872088    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vlwq\" (UniqueName: \"kubernetes.io/projected/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-kube-api-access-9vlwq\") pod \"limitador-limitador-859c4d96d6-mks6d\" (UID: \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:26.872273 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.872159    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-config-file\") pod \"limitador-limitador-859c4d96d6-mks6d\" (UID: \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:26.905745 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.905709    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-mks6d"]
Jul 29 10:38:26.913648 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.913622    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:26.972957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.972928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vlwq\" (UniqueName: \"kubernetes.io/projected/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-kube-api-access-9vlwq\") pod \"limitador-limitador-859c4d96d6-mks6d\" (UID: \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:26.973094 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.972966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-config-file\") pod \"limitador-limitador-859c4d96d6-mks6d\" (UID: \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:26.973559 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.973542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-config-file\") pod \"limitador-limitador-859c4d96d6-mks6d\" (UID: \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:26.976214 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.976189    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:26.981141 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:26.981122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vlwq\" (UniqueName: \"kubernetes.io/projected/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-kube-api-access-9vlwq\") pod \"limitador-limitador-859c4d96d6-mks6d\" (UID: \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:27.131510 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:27.131477    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:27.267070 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:27.267049    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-mks6d"]
Jul 29 10:38:27.269093 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:38:27.269067    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a88e1c_20dd_4bce_91fc_d3aa9d55191e.slice/crio-4e26f01894c7dfce95b763636bd7d38dd50bf2d50755a76f8b51691f21daa1b2 WatchSource:0}: Error finding container 4e26f01894c7dfce95b763636bd7d38dd50bf2d50755a76f8b51691f21daa1b2: Status 404 returned error can't find the container with id 4e26f01894c7dfce95b763636bd7d38dd50bf2d50755a76f8b51691f21daa1b2
Jul 29 10:38:27.471363 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:27.471285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d" event={"ID":"04a88e1c-20dd-4bce-91fc-d3aa9d55191e","Type":"ContainerStarted","Data":"4e26f01894c7dfce95b763636bd7d38dd50bf2d50755a76f8b51691f21daa1b2"}
Jul 29 10:38:27.643077 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:27.643040    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:27.709770 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:27.709729    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:31.490461 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:31.490415    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d" event={"ID":"04a88e1c-20dd-4bce-91fc-d3aa9d55191e","Type":"ContainerStarted","Data":"63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677"}
Jul 29 10:38:31.490881 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:31.490542    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:31.508933 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:31.508879    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d" podStartSLOduration=2.33619185 podStartE2EDuration="5.508863927s" totalImagesPullingTime="3.172672077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:31.5071996 +0000 UTC m=+585.900412490" watchObservedRunningTime="2026-07-29 10:38:31.508863927 +0000 UTC m=+585.902076796"
Jul 29 10:38:42.130445 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.130393    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-mks6d"]
Jul 29 10:38:42.130940 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.130611    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d" podUID="04a88e1c-20dd-4bce-91fc-d3aa9d55191e" containerName="limitador" containerID="cri-o://63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677" gracePeriod=30
Jul 29 10:38:42.131305 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.131285    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:42.678456 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.678417    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:42.816364 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.816292    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-config-file\") pod \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\" (UID: \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\") "
Jul 29 10:38:42.816364 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.816357    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vlwq\" (UniqueName: \"kubernetes.io/projected/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-kube-api-access-9vlwq\") pod \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\" (UID: \"04a88e1c-20dd-4bce-91fc-d3aa9d55191e\") "
Jul 29 10:38:42.816656 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.816630    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-config-file" (OuterVolumeSpecName: "config-file") pod "04a88e1c-20dd-4bce-91fc-d3aa9d55191e" (UID: "04a88e1c-20dd-4bce-91fc-d3aa9d55191e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:38:42.818525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.818491    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-kube-api-access-9vlwq" (OuterVolumeSpecName: "kube-api-access-9vlwq") pod "04a88e1c-20dd-4bce-91fc-d3aa9d55191e" (UID: "04a88e1c-20dd-4bce-91fc-d3aa9d55191e"). InnerVolumeSpecName "kube-api-access-9vlwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:38:42.917446 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.917405    2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-config-file\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:38:42.917446 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:42.917449    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vlwq\" (UniqueName: \"kubernetes.io/projected/04a88e1c-20dd-4bce-91fc-d3aa9d55191e-kube-api-access-9vlwq\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:38:43.258166 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.258128    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:43.546226 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.546136    2569 generic.go:356] "Generic (PLEG): container finished" podID="04a88e1c-20dd-4bce-91fc-d3aa9d55191e" containerID="63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677" exitCode=0
Jul 29 10:38:43.546226 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.546201    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d"
Jul 29 10:38:43.546480 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.546224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d" event={"ID":"04a88e1c-20dd-4bce-91fc-d3aa9d55191e","Type":"ContainerDied","Data":"63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677"}
Jul 29 10:38:43.546480 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.546288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-mks6d" event={"ID":"04a88e1c-20dd-4bce-91fc-d3aa9d55191e","Type":"ContainerDied","Data":"4e26f01894c7dfce95b763636bd7d38dd50bf2d50755a76f8b51691f21daa1b2"}
Jul 29 10:38:43.546480 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.546318    2569 scope.go:117] "RemoveContainer" containerID="63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677"
Jul 29 10:38:43.554783 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.554766    2569 scope.go:117] "RemoveContainer" containerID="63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677"
Jul 29 10:38:43.555035 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:38:43.555017    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677\": container with ID starting with 63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677 not found: ID does not exist" containerID="63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677"
Jul 29 10:38:43.555104 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.555048    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677"} err="failed to get container status \"63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677\": rpc error: code = NotFound desc = could not find container \"63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677\": container with ID starting with 63883cb6351000551c4545540567041b791d5e9cb51d60fcdd65bed58ba99677 not found: ID does not exist"
Jul 29 10:38:43.568047 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.568019    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-mks6d"]
Jul 29 10:38:43.570562 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:43.570540    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-mks6d"]
Jul 29 10:38:44.041771 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:44.041735    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a88e1c-20dd-4bce-91fc-d3aa9d55191e" path="/var/lib/kubelet/pods/04a88e1c-20dd-4bce-91fc-d3aa9d55191e/volumes"
Jul 29 10:38:48.084550 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:48.084515    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:48.833945 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:48.833913    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:58.914653 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:58.914619    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:58.985661 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:58.985627    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:59.067914 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:59.067874    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:38:59.168688 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:38:59.168572    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:02.453462 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:02.453398    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:04.051713 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.051678    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-65f566c74b-v6w4s"]
Jul 29 10:39:04.052116 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.051866    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s" podUID="7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d" containerName="manager" containerID="cri-o://596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056" gracePeriod=10
Jul 29 10:39:04.073869 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.073845    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-65f566c74b-m6f8j"]
Jul 29 10:39:04.074220 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.074207    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a88e1c-20dd-4bce-91fc-d3aa9d55191e" containerName="limitador"
Jul 29 10:39:04.074262 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.074222    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a88e1c-20dd-4bce-91fc-d3aa9d55191e" containerName="limitador"
Jul 29 10:39:04.074296 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.074276    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a88e1c-20dd-4bce-91fc-d3aa9d55191e" containerName="limitador"
Jul 29 10:39:04.077776 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.077758    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:04.087934 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.087904    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-65f566c74b-m6f8j"]
Jul 29 10:39:04.109315 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.109286    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f515603-6f8c-47f7-a49f-9f0ed77eaee1-cert\") pod \"kserve-controller-manager-65f566c74b-m6f8j\" (UID: \"6f515603-6f8c-47f7-a49f-9f0ed77eaee1\") " pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:04.109468 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.109380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnhj\" (UniqueName: \"kubernetes.io/projected/6f515603-6f8c-47f7-a49f-9f0ed77eaee1-kube-api-access-vhnhj\") pod \"kserve-controller-manager-65f566c74b-m6f8j\" (UID: \"6f515603-6f8c-47f7-a49f-9f0ed77eaee1\") " pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:04.210301 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.210269    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f515603-6f8c-47f7-a49f-9f0ed77eaee1-cert\") pod \"kserve-controller-manager-65f566c74b-m6f8j\" (UID: \"6f515603-6f8c-47f7-a49f-9f0ed77eaee1\") " pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:04.210474 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.210330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnhj\" (UniqueName: \"kubernetes.io/projected/6f515603-6f8c-47f7-a49f-9f0ed77eaee1-kube-api-access-vhnhj\") pod \"kserve-controller-manager-65f566c74b-m6f8j\" (UID: \"6f515603-6f8c-47f7-a49f-9f0ed77eaee1\") " pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:04.212639 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.212609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f515603-6f8c-47f7-a49f-9f0ed77eaee1-cert\") pod \"kserve-controller-manager-65f566c74b-m6f8j\" (UID: \"6f515603-6f8c-47f7-a49f-9f0ed77eaee1\") " pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:04.220541 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.220514    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnhj\" (UniqueName: \"kubernetes.io/projected/6f515603-6f8c-47f7-a49f-9f0ed77eaee1-kube-api-access-vhnhj\") pod \"kserve-controller-manager-65f566c74b-m6f8j\" (UID: \"6f515603-6f8c-47f7-a49f-9f0ed77eaee1\") " pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:04.314966 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.314912    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:39:04.411280 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.411250    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqgkw\" (UniqueName: \"kubernetes.io/projected/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-kube-api-access-wqgkw\") pod \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\" (UID: \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\") "
Jul 29 10:39:04.411472 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.411308    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-cert\") pod \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\" (UID: \"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d\") "
Jul 29 10:39:04.413345 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.413320    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-kube-api-access-wqgkw" (OuterVolumeSpecName: "kube-api-access-wqgkw") pod "7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d" (UID: "7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d"). InnerVolumeSpecName "kube-api-access-wqgkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:39:04.413345 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.413334    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-cert" (OuterVolumeSpecName: "cert") pod "7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d" (UID: "7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:39:04.426554 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.426536    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:04.512012 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.511982    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqgkw\" (UniqueName: \"kubernetes.io/projected/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-kube-api-access-wqgkw\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:39:04.512012 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.512012    2569 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:39:04.558227 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.558193    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-65f566c74b-m6f8j"]
Jul 29 10:39:04.648489 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.648454    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-65f566c74b-m6f8j" event={"ID":"6f515603-6f8c-47f7-a49f-9f0ed77eaee1","Type":"ContainerStarted","Data":"e84f8408a734eec727a8445554293aa6fa925e0af23e6cddc20600e3b663d7c2"}
Jul 29 10:39:04.649712 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.649685    2569 generic.go:356] "Generic (PLEG): container finished" podID="7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d" containerID="596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056" exitCode=0
Jul 29 10:39:04.649848 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.649744    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s"
Jul 29 10:39:04.649848 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.649748    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s" event={"ID":"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d","Type":"ContainerDied","Data":"596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056"}
Jul 29 10:39:04.649959 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.649865    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-65f566c74b-v6w4s" event={"ID":"7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d","Type":"ContainerDied","Data":"1566633c096c3e331b532d332d4508e4102556987920f449a5e7869a647e95d3"}
Jul 29 10:39:04.649959 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.649898    2569 scope.go:117] "RemoveContainer" containerID="596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056"
Jul 29 10:39:04.658844 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.658826    2569 scope.go:117] "RemoveContainer" containerID="596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056"
Jul 29 10:39:04.659111 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:39:04.659088    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056\": container with ID starting with 596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056 not found: ID does not exist" containerID="596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056"
Jul 29 10:39:04.659200 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.659117    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056"} err="failed to get container status \"596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056\": rpc error: code = NotFound desc = could not find container \"596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056\": container with ID starting with 596d6d6b1065ce566ec2a6c3a725e3527c11d29328c980c5a54bc599230de056 not found: ID does not exist"
Jul 29 10:39:04.672203 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.672178    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-65f566c74b-v6w4s"]
Jul 29 10:39:04.678009 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:04.677988    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-65f566c74b-v6w4s"]
Jul 29 10:39:05.658541 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:05.658506    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-65f566c74b-m6f8j" event={"ID":"6f515603-6f8c-47f7-a49f-9f0ed77eaee1","Type":"ContainerStarted","Data":"49fec6840f9d7d0b1e8609d742ff2a2275c7a3da0102c4f43870a4ba6f5306fe"}
Jul 29 10:39:05.658912 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:05.658625    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:05.675971 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:05.675924    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-65f566c74b-m6f8j" podStartSLOduration=1.18492666 podStartE2EDuration="1.675910041s" totalImagesPullingTime="490.983381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:05.674903379 +0000 UTC m=+620.068116247" watchObservedRunningTime="2026-07-29 10:39:05.675910041 +0000 UTC m=+620.069122910"
Jul 29 10:39:06.041482 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:06.041394    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d" path="/var/lib/kubelet/pods/7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d/volumes"
Jul 29 10:39:20.322913 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:20.322866    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:36.668243 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:36.668212    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-65f566c74b-m6f8j"
Jul 29 10:39:37.634918 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.634884    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:37.669292 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.669255    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-bj279"]
Jul 29 10:39:37.669668 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.669648    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d" containerName="manager"
Jul 29 10:39:37.669668 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.669662    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d" containerName="manager"
Jul 29 10:39:37.669778 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.669766    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b9611c2-d4b3-4945-8b28-eb1af6cbaa6d" containerName="manager"
Jul 29 10:39:37.673121 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.673100    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:37.675728 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.675700    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 29 10:39:37.675861 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.675750    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-cj75q\""
Jul 29 10:39:37.680911 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.680884    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-6h4sv"]
Jul 29 10:39:37.684671 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.684648    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:37.685370 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.685347    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bj279"]
Jul 29 10:39:37.687550 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.687529    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 29 10:39:37.687752 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.687600    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-ngsgf\""
Jul 29 10:39:37.693140 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.693118    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-6h4sv"]
Jul 29 10:39:37.725391 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.725359    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:37.796984 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.796952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qvn\" (UniqueName: \"kubernetes.io/projected/3f50e258-57c9-4278-a06d-25d8c49a8719-kube-api-access-t9qvn\") pod \"odh-model-controller-75d9cdbddf-6h4sv\" (UID: \"3f50e258-57c9-4278-a06d-25d8c49a8719\") " pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:37.797185 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.797015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/918aa4b5-19e1-4fee-89a8-92fef107680e-tls-certs\") pod \"model-serving-api-7b954ffc95-bj279\" (UID: \"918aa4b5-19e1-4fee-89a8-92fef107680e\") " pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:37.797185 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.797109    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f50e258-57c9-4278-a06d-25d8c49a8719-cert\") pod \"odh-model-controller-75d9cdbddf-6h4sv\" (UID: \"3f50e258-57c9-4278-a06d-25d8c49a8719\") " pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:37.797277 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.797178    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mknhn\" (UniqueName: \"kubernetes.io/projected/918aa4b5-19e1-4fee-89a8-92fef107680e-kube-api-access-mknhn\") pod \"model-serving-api-7b954ffc95-bj279\" (UID: \"918aa4b5-19e1-4fee-89a8-92fef107680e\") " pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:37.826739 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.826705    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:37.898255 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.898163    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mknhn\" (UniqueName: \"kubernetes.io/projected/918aa4b5-19e1-4fee-89a8-92fef107680e-kube-api-access-mknhn\") pod \"model-serving-api-7b954ffc95-bj279\" (UID: \"918aa4b5-19e1-4fee-89a8-92fef107680e\") " pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:37.898255 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.898213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qvn\" (UniqueName: \"kubernetes.io/projected/3f50e258-57c9-4278-a06d-25d8c49a8719-kube-api-access-t9qvn\") pod \"odh-model-controller-75d9cdbddf-6h4sv\" (UID: \"3f50e258-57c9-4278-a06d-25d8c49a8719\") " pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:37.898545 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.898261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/918aa4b5-19e1-4fee-89a8-92fef107680e-tls-certs\") pod \"model-serving-api-7b954ffc95-bj279\" (UID: \"918aa4b5-19e1-4fee-89a8-92fef107680e\") " pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:37.898545 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.898330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f50e258-57c9-4278-a06d-25d8c49a8719-cert\") pod \"odh-model-controller-75d9cdbddf-6h4sv\" (UID: \"3f50e258-57c9-4278-a06d-25d8c49a8719\") " pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:37.898545 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:39:37.898488    2569 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 29 10:39:37.898701 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:39:37.898577    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f50e258-57c9-4278-a06d-25d8c49a8719-cert podName:3f50e258-57c9-4278-a06d-25d8c49a8719 nodeName:}" failed. No retries permitted until 2026-07-29 10:39:38.398554579 +0000 UTC m=+652.791767427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3f50e258-57c9-4278-a06d-25d8c49a8719-cert") pod "odh-model-controller-75d9cdbddf-6h4sv" (UID: "3f50e258-57c9-4278-a06d-25d8c49a8719") : secret "odh-model-controller-webhook-cert" not found
Jul 29 10:39:37.900796 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.900764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/918aa4b5-19e1-4fee-89a8-92fef107680e-tls-certs\") pod \"model-serving-api-7b954ffc95-bj279\" (UID: \"918aa4b5-19e1-4fee-89a8-92fef107680e\") " pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:37.909452 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.909400    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qvn\" (UniqueName: \"kubernetes.io/projected/3f50e258-57c9-4278-a06d-25d8c49a8719-kube-api-access-t9qvn\") pod \"odh-model-controller-75d9cdbddf-6h4sv\" (UID: \"3f50e258-57c9-4278-a06d-25d8c49a8719\") " pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:37.909641 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.909621    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mknhn\" (UniqueName: \"kubernetes.io/projected/918aa4b5-19e1-4fee-89a8-92fef107680e-kube-api-access-mknhn\") pod \"model-serving-api-7b954ffc95-bj279\" (UID: \"918aa4b5-19e1-4fee-89a8-92fef107680e\") " pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:37.979962 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.979926    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:37.987279 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:37.987252    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:38.121610 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:38.121575    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bj279"]
Jul 29 10:39:38.123916 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:39:38.123884    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918aa4b5_19e1_4fee_89a8_92fef107680e.slice/crio-6ef78babf62aa726835900ad9fc84a06fcdfc5125c75e0deb6c37ffc11d32db0 WatchSource:0}: Error finding container 6ef78babf62aa726835900ad9fc84a06fcdfc5125c75e0deb6c37ffc11d32db0: Status 404 returned error can't find the container with id 6ef78babf62aa726835900ad9fc84a06fcdfc5125c75e0deb6c37ffc11d32db0
Jul 29 10:39:38.271738 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:38.271653    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:38.327588 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:38.327554    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:39:38.401783 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:38.401748    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f50e258-57c9-4278-a06d-25d8c49a8719-cert\") pod \"odh-model-controller-75d9cdbddf-6h4sv\" (UID: \"3f50e258-57c9-4278-a06d-25d8c49a8719\") " pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:38.404129 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:38.404112    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f50e258-57c9-4278-a06d-25d8c49a8719-cert\") pod \"odh-model-controller-75d9cdbddf-6h4sv\" (UID: \"3f50e258-57c9-4278-a06d-25d8c49a8719\") " pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:38.598375 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:38.598340    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:38.730818 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:38.730788    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-6h4sv"]
Jul 29 10:39:38.732922 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:39:38.732892    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f50e258_57c9_4278_a06d_25d8c49a8719.slice/crio-86e9eeaaffe9ffe2268539bc75cac1e964d4d9a5642b3a9a3efc0770cc41d7a2 WatchSource:0}: Error finding container 86e9eeaaffe9ffe2268539bc75cac1e964d4d9a5642b3a9a3efc0770cc41d7a2: Status 404 returned error can't find the container with id 86e9eeaaffe9ffe2268539bc75cac1e964d4d9a5642b3a9a3efc0770cc41d7a2
Jul 29 10:39:38.813438 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:38.813393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-6h4sv" event={"ID":"3f50e258-57c9-4278-a06d-25d8c49a8719","Type":"ContainerStarted","Data":"86e9eeaaffe9ffe2268539bc75cac1e964d4d9a5642b3a9a3efc0770cc41d7a2"}
Jul 29 10:39:38.814742 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:38.814701    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bj279" event={"ID":"918aa4b5-19e1-4fee-89a8-92fef107680e","Type":"ContainerStarted","Data":"6ef78babf62aa726835900ad9fc84a06fcdfc5125c75e0deb6c37ffc11d32db0"}
Jul 29 10:39:40.828891 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:40.828855    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bj279" event={"ID":"918aa4b5-19e1-4fee-89a8-92fef107680e","Type":"ContainerStarted","Data":"ce3bf61ce4b86c4f65bf1901f49248106d92f36b688931504586c6929e703d15"}
Jul 29 10:39:40.829280 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:40.828921    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:40.846095 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:40.846045    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-bj279" podStartSLOduration=1.948032013 podStartE2EDuration="3.846029428s" totalImagesPullingTime="1.897997415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:40.844961474 +0000 UTC m=+655.238174344" watchObservedRunningTime="2026-07-29 10:39:40.846029428 +0000 UTC m=+655.239242326"
Jul 29 10:39:41.835894 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:41.835860    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-6h4sv" event={"ID":"3f50e258-57c9-4278-a06d-25d8c49a8719","Type":"ContainerStarted","Data":"2903ddaa853f0cff3283aa448821a63eaa5210eb342e397ca721a5226f96d5c4"}
Jul 29 10:39:41.836380 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:41.835926    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:39:41.854221 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:41.854171    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-6h4sv" podStartSLOduration=2.200772684 podStartE2EDuration="4.854153936s" totalImagesPullingTime="2.653381252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:41.852317371 +0000 UTC m=+656.245530242" watchObservedRunningTime="2026-07-29 10:39:41.854153936 +0000 UTC m=+656.247366807"
Jul 29 10:39:51.839989 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:51.839960    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-bj279"
Jul 29 10:39:52.844819 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:39:52.844789    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-6h4sv"
Jul 29 10:40:33.558293 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:40:33.558246    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:13.296751 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:13.296715    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:13.396615 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:13.396586    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:25.855253 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:25.855215    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:25.947901 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:25.947872    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:28.836406 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:28.836364    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:29.195687 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.195617    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"]
Jul 29 10:41:29.199092 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.199072    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:41:29.202964 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.202943    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 29 10:41:29.203089 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.202992    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 29 10:41:29.203089 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.203009    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-r8s7v\""
Jul 29 10:41:29.203089 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.202946    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 29 10:41:29.203259 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.203244    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-gxgdc\""
Jul 29 10:41:29.207628 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.207605    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"]
Jul 29 10:41:29.223254 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.223222    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:29.301317 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.301284    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fp8x\" (UniqueName: \"kubernetes.io/projected/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-kube-api-access-4fp8x\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5\" (UID: \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:41:29.301514 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.301341    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5\" (UID: \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:41:29.402215 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.402179    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fp8x\" (UniqueName: \"kubernetes.io/projected/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-kube-api-access-4fp8x\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5\" (UID: \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:41:29.402404 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.402231    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5\" (UID: \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:41:29.404954 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.404928    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5\" (UID: \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:41:29.410942 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.410913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fp8x\" (UniqueName: \"kubernetes.io/projected/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-kube-api-access-4fp8x\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5\" (UID: \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:41:29.510190 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.510099    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:41:29.653660 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.653637    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"]
Jul 29 10:41:29.656162 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:41:29.656136    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ee13bf_452f_4e99_bd3e_4925b5c12c4c.slice/crio-52d2c7a1d566bf753dfee676b1f7e38d5d4d86d6f8d8970a88201048b27238c8 WatchSource:0}: Error finding container 52d2c7a1d566bf753dfee676b1f7e38d5d4d86d6f8d8970a88201048b27238c8: Status 404 returned error can't find the container with id 52d2c7a1d566bf753dfee676b1f7e38d5d4d86d6f8d8970a88201048b27238c8
Jul 29 10:41:29.658230 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:29.658212    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:41:30.342796 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:30.342753    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5" event={"ID":"78ee13bf-452f-4e99-bd3e-4925b5c12c4c","Type":"ContainerStarted","Data":"52d2c7a1d566bf753dfee676b1f7e38d5d4d86d6f8d8970a88201048b27238c8"}
Jul 29 10:41:32.197204 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:32.197174    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:32.356564 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:32.356515    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5" event={"ID":"78ee13bf-452f-4e99-bd3e-4925b5c12c4c","Type":"ContainerStarted","Data":"ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8"}
Jul 29 10:41:32.356817 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:32.356656    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:41:32.374122 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:32.374065    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5" podStartSLOduration=0.858401784 podStartE2EDuration="3.374043894s" totalImagesPullingTime="2.51564211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:41:32.372437197 +0000 UTC m=+766.765650082" watchObservedRunningTime="2026-07-29 10:41:32.374043894 +0000 UTC m=+766.767256765"
Jul 29 10:41:40.243123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:40.243086    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:41:56.543203 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:41:56.543170    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:03.363286 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:03.363256    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:42:09.496317 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:09.496243    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:09.496752 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:09.496676    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"]
Jul 29 10:42:09.608840 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:09.608808    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:13.362984 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:13.362949    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5" podUID="78ee13bf-452f-4e99-bd3e-4925b5c12c4c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:42:23.363512 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:23.363475    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5" podUID="78ee13bf-452f-4e99-bd3e-4925b5c12c4c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:42:24.516355 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:24.516291    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5" podUID="78ee13bf-452f-4e99-bd3e-4925b5c12c4c" containerName="main" containerID="cri-o://ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8" gracePeriod=45
Jul 29 10:42:24.762192 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:24.762166    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:42:24.797441 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:24.797346    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-tls-certs\") pod \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\" (UID: \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\") "
Jul 29 10:42:24.797441 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:24.797412    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fp8x\" (UniqueName: \"kubernetes.io/projected/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-kube-api-access-4fp8x\") pod \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\" (UID: \"78ee13bf-452f-4e99-bd3e-4925b5c12c4c\") "
Jul 29 10:42:24.799527 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:24.799491    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78ee13bf-452f-4e99-bd3e-4925b5c12c4c" (UID: "78ee13bf-452f-4e99-bd3e-4925b5c12c4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:42:24.799656 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:24.799610    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-kube-api-access-4fp8x" (OuterVolumeSpecName: "kube-api-access-4fp8x") pod "78ee13bf-452f-4e99-bd3e-4925b5c12c4c" (UID: "78ee13bf-452f-4e99-bd3e-4925b5c12c4c"). InnerVolumeSpecName "kube-api-access-4fp8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:42:24.898459 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:24.898396    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:42:24.898459 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:24.898447    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fp8x\" (UniqueName: \"kubernetes.io/projected/78ee13bf-452f-4e99-bd3e-4925b5c12c4c-kube-api-access-4fp8x\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:42:25.610394 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:25.610358    2569 generic.go:356] "Generic (PLEG): container finished" podID="78ee13bf-452f-4e99-bd3e-4925b5c12c4c" containerID="ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8" exitCode=0
Jul 29 10:42:25.610394 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:25.610397    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"
Jul 29 10:42:25.610919 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:25.610461    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5" event={"ID":"78ee13bf-452f-4e99-bd3e-4925b5c12c4c","Type":"ContainerDied","Data":"ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8"}
Jul 29 10:42:25.610919 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:25.610519    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5" event={"ID":"78ee13bf-452f-4e99-bd3e-4925b5c12c4c","Type":"ContainerDied","Data":"52d2c7a1d566bf753dfee676b1f7e38d5d4d86d6f8d8970a88201048b27238c8"}
Jul 29 10:42:25.610919 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:25.610545    2569 scope.go:117] "RemoveContainer" containerID="ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8"
Jul 29 10:42:25.618954 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:25.618939    2569 scope.go:117] "RemoveContainer" containerID="ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8"
Jul 29 10:42:25.619208 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:42:25.619189    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8\": container with ID starting with ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8 not found: ID does not exist" containerID="ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8"
Jul 29 10:42:25.619254 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:25.619216    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8"} err="failed to get container status \"ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8\": rpc error: code = NotFound desc = could not find container \"ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8\": container with ID starting with ec95ac4d96379434e98de5878304d96bb9d6c5041be20accf5224354a02321a8 not found: ID does not exist"
Jul 29 10:42:25.631770 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:25.631745    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"]
Jul 29 10:42:25.635395 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:25.635372    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-mskb5"]
Jul 29 10:42:26.042156 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:26.042083    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78ee13bf-452f-4e99-bd3e-4925b5c12c4c" path="/var/lib/kubelet/pods/78ee13bf-452f-4e99-bd3e-4925b5c12c4c/volumes"
Jul 29 10:42:29.446519 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:29.446469    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:29.544369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:29.544334    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:35.724494 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:35.724423    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:41.199664 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:41.199633    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:41.262675 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:41.262646    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:42.160392 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.160360    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:42.327165 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.327135    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"]
Jul 29 10:42:42.327613 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.327569    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ee13bf-452f-4e99-bd3e-4925b5c12c4c" containerName="main"
Jul 29 10:42:42.327613 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.327587    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ee13bf-452f-4e99-bd3e-4925b5c12c4c" containerName="main"
Jul 29 10:42:42.327817 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.327720    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ee13bf-452f-4e99-bd3e-4925b5c12c4c" containerName="main"
Jul 29 10:42:42.330610 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.330593    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:42:42.334605 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.334585    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 29 10:42:42.334706 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.334616    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 29 10:42:42.334706 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.334657    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-nv9w9\""
Jul 29 10:42:42.334805 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.334760    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 29 10:42:42.334849 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.334837    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-klv5f\""
Jul 29 10:42:42.339028 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.339010    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"]
Jul 29 10:42:42.452465 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.452346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm\" (UID: \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:42:42.452465 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.452408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfl5c\" (UniqueName: \"kubernetes.io/projected/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-kube-api-access-tfl5c\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm\" (UID: \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:42:42.517203 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.517171    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:42:42.553719 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.553687    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfl5c\" (UniqueName: \"kubernetes.io/projected/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-kube-api-access-tfl5c\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm\" (UID: \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:42:42.553889 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.553772    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm\" (UID: \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:42:42.556194 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.556171    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm\" (UID: \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:42:42.563818 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.563796    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfl5c\" (UniqueName: \"kubernetes.io/projected/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-kube-api-access-tfl5c\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm\" (UID: \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:42:42.641590 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.641554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:42:42.782207 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:42.782183    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"]
Jul 29 10:42:42.784448 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:42:42.784397    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8353cb_d05b_47a6_9339_dcc4ec5e9397.slice/crio-e881c70f057abe6789631cdf2ef76cf2dad5b44736903288c7bc650045801f88 WatchSource:0}: Error finding container e881c70f057abe6789631cdf2ef76cf2dad5b44736903288c7bc650045801f88: Status 404 returned error can't find the container with id e881c70f057abe6789631cdf2ef76cf2dad5b44736903288c7bc650045801f88
Jul 29 10:42:43.701910 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:43.701871    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm" event={"ID":"1d8353cb-d05b-47a6-9339-dcc4ec5e9397","Type":"ContainerStarted","Data":"d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1"}
Jul 29 10:42:43.702369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:43.701930    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:42:43.702369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:43.701946    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm" event={"ID":"1d8353cb-d05b-47a6-9339-dcc4ec5e9397","Type":"ContainerStarted","Data":"e881c70f057abe6789631cdf2ef76cf2dad5b44736903288c7bc650045801f88"}
Jul 29 10:42:43.719690 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:43.719643    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm" podStartSLOduration=1.7196275170000002 podStartE2EDuration="1.719627517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:42:43.719114828 +0000 UTC m=+838.112327700" watchObservedRunningTime="2026-07-29 10:42:43.719627517 +0000 UTC m=+838.112840387"
Jul 29 10:42:53.468886 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:42:53.468851    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:14.708719 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:14.708690    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:43:21.356387 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:21.356349    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"]
Jul 29 10:43:21.365080 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:21.365050    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:21.429227 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:21.429190    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:24.709191 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:24.709157    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm" podUID="1d8353cb-d05b-47a6-9339-dcc4ec5e9397" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:43:34.708805 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:34.708761    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm" podUID="1d8353cb-d05b-47a6-9339-dcc4ec5e9397" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:43:36.380204 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.380162    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm" podUID="1d8353cb-d05b-47a6-9339-dcc4ec5e9397" containerName="main" containerID="cri-o://d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1" gracePeriod=45
Jul 29 10:43:36.629369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.629338    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:43:36.710485 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.710377    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-tls-certs\") pod \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\" (UID: \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\") "
Jul 29 10:43:36.710485 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.710463    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfl5c\" (UniqueName: \"kubernetes.io/projected/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-kube-api-access-tfl5c\") pod \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\" (UID: \"1d8353cb-d05b-47a6-9339-dcc4ec5e9397\") "
Jul 29 10:43:36.712625 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.712594    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d8353cb-d05b-47a6-9339-dcc4ec5e9397" (UID: "1d8353cb-d05b-47a6-9339-dcc4ec5e9397"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:43:36.712625 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.712610    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-kube-api-access-tfl5c" (OuterVolumeSpecName: "kube-api-access-tfl5c") pod "1d8353cb-d05b-47a6-9339-dcc4ec5e9397" (UID: "1d8353cb-d05b-47a6-9339-dcc4ec5e9397"). InnerVolumeSpecName "kube-api-access-tfl5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:43:36.811732 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.811701    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfl5c\" (UniqueName: \"kubernetes.io/projected/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-kube-api-access-tfl5c\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:43:36.811732 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.811728    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8353cb-d05b-47a6-9339-dcc4ec5e9397-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:43:36.956724 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.956690    2569 generic.go:356] "Generic (PLEG): container finished" podID="1d8353cb-d05b-47a6-9339-dcc4ec5e9397" containerID="d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1" exitCode=0
Jul 29 10:43:36.956724 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.956726    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"
Jul 29 10:43:36.956919 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.956765    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm" event={"ID":"1d8353cb-d05b-47a6-9339-dcc4ec5e9397","Type":"ContainerDied","Data":"d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1"}
Jul 29 10:43:36.956919 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.956819    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm" event={"ID":"1d8353cb-d05b-47a6-9339-dcc4ec5e9397","Type":"ContainerDied","Data":"e881c70f057abe6789631cdf2ef76cf2dad5b44736903288c7bc650045801f88"}
Jul 29 10:43:36.956919 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.956848    2569 scope.go:117] "RemoveContainer" containerID="d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1"
Jul 29 10:43:36.966783 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.966764    2569 scope.go:117] "RemoveContainer" containerID="d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1"
Jul 29 10:43:36.967083 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:43:36.967037    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1\": container with ID starting with d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1 not found: ID does not exist" containerID="d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1"
Jul 29 10:43:36.967138 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.967092    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1"} err="failed to get container status \"d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1\": rpc error: code = NotFound desc = could not find container \"d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1\": container with ID starting with d4b998c41bcea5b82f77782a25dc33ed72a5ffc8af2ac048d93dcc9516a14ad1 not found: ID does not exist"
Jul 29 10:43:36.978733 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.978704    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"]
Jul 29 10:43:36.982360 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:36.982336    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-8f2qm"]
Jul 29 10:43:38.042099 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:38.042066    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d8353cb-d05b-47a6-9339-dcc4ec5e9397" path="/var/lib/kubelet/pods/1d8353cb-d05b-47a6-9339-dcc4ec5e9397/volumes"
Jul 29 10:43:41.373076 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:41.373040    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:41.472942 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:41.472912    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:41.653340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:41.653254    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:41.758087 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:41.758051    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:45.129905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:45.129848    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:47.785610 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:47.785570    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:51.443665 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:51.443621    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:53.516272 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:53.516217    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:53.616646 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:53.616618    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:53.851769 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:53.851740    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:54.050444 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.050411    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:54.249976 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.249896    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:54.336283 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.336211    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"]
Jul 29 10:43:54.336946 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.336919    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8353cb-d05b-47a6-9339-dcc4ec5e9397" containerName="main"
Jul 29 10:43:54.336946 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.336946    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8353cb-d05b-47a6-9339-dcc4ec5e9397" containerName="main"
Jul 29 10:43:54.337129 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.337074    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8353cb-d05b-47a6-9339-dcc4ec5e9397" containerName="main"
Jul 29 10:43:54.341647 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.341628    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.344946 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.344921    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 29 10:43:54.345052 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.345014    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-lpsln\""
Jul 29 10:43:54.345409 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.345395    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 29 10:43:54.346225 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.346210    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 29 10:43:54.352054 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.352033    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"]
Jul 29 10:43:54.378634 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.378599    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:54.472236 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.472198    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.472470 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.472246    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.472470 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.472304    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.472470 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.472357    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa95b76-571d-45ae-8e5a-560db0dbf891-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.472470 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.472394    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xzl\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-kube-api-access-v6xzl\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.472470 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.472460    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.472693 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.472518    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.472693 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.472535    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.472693 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.472612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.477120 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.477097    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:54.573392 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573306    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.573392 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.573957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573406    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.573957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573454    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.573957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa95b76-571d-45ae-8e5a-560db0dbf891-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.573957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573533    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xzl\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-kube-api-access-v6xzl\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.573957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.573957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.573957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.573957 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573827    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.574374 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.573978    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.574374 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.574023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.574374 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.574033    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.574374 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.574318    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa95b76-571d-45ae-8e5a-560db0dbf891-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.575854 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.575834    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.576216 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.576195    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.582270 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.582246    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.582367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.582301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xzl\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-kube-api-access-v6xzl\") pod \"router-gateway-1-openshift-default-66bbccc5fb-7npr9\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.652994 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.652964    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:54.794100 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:54.794076    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"]
Jul 29 10:43:54.796315 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:43:54.796284    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa95b76_571d_45ae_8e5a_560db0dbf891.slice/crio-ce65ce2f18b6ed5e7e1570f3015bba5df02af5432f62a29864c0bfa28ef151fe WatchSource:0}: Error finding container ce65ce2f18b6ed5e7e1570f3015bba5df02af5432f62a29864c0bfa28ef151fe: Status 404 returned error can't find the container with id ce65ce2f18b6ed5e7e1570f3015bba5df02af5432f62a29864c0bfa28ef151fe
Jul 29 10:43:55.045004 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:55.044964    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9" event={"ID":"caa95b76-571d-45ae-8e5a-560db0dbf891","Type":"ContainerStarted","Data":"ce65ce2f18b6ed5e7e1570f3015bba5df02af5432f62a29864c0bfa28ef151fe"}
Jul 29 10:43:57.955622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:57.955578    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:57.994830 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:57.994784    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:43:57.994959 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:57.994869    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:43:57.994959 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:57.994899    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 29 10:43:58.064194 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:58.064166    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9" event={"ID":"caa95b76-571d-45ae-8e5a-560db0dbf891","Type":"ContainerStarted","Data":"2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203"}
Jul 29 10:43:58.085971 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:58.085900    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9" podStartSLOduration=0.889601029 podStartE2EDuration="4.085876133s" totalImagesPullingTime="3.196275104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:43:58.08525497 +0000 UTC m=+912.478467888" watchObservedRunningTime="2026-07-29 10:43:58.085876133 +0000 UTC m=+912.479089004"
Jul 29 10:43:58.653214 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:58.653177    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:43:58.879939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:58.879908    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:59.123258 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:59.123226    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:43:59.659609 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:43:59.659577    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:44:00.075304 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:00.075211    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:44:00.106857 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:00.106828    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:44:01.346312 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:01.346277    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:01.437598 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:01.437568    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:01.832932 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:01.832901    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:03.954593 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:03.954563    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:05.021948 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:05.021831    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:06.317461 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:06.317412    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"]
Jul 29 10:44:06.317940 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:06.317626    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9" podUID="caa95b76-571d-45ae-8e5a-560db0dbf891" containerName="istio-proxy" containerID="cri-o://2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203" gracePeriod=30
Jul 29 10:44:06.368859 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:06.368828    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:08.051448 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.051368    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:08.237390 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.237360    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"]
Jul 29 10:44:08.241372 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.241351    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:08.245751 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.245721    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 29 10:44:08.245890 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.245733    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-ncs6p\""
Jul 29 10:44:08.245890 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.245859    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 29 10:44:08.246020 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.245901    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 29 10:44:08.246020 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.245945    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-ln9z9\""
Jul 29 10:44:08.253745 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.253726    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"]
Jul 29 10:44:08.270706 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.270671    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:08.308146 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.308071    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r69w\" (UniqueName: \"kubernetes.io/projected/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-kube-api-access-9r69w\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69\" (UID: \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:08.308146 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.308140    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69\" (UID: \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:08.408818 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.408780    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69\" (UID: \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:08.408972 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.408859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r69w\" (UniqueName: \"kubernetes.io/projected/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-kube-api-access-9r69w\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69\" (UID: \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:08.411347 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.411322    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69\" (UID: \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:08.417344 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.417307    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r69w\" (UniqueName: \"kubernetes.io/projected/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-kube-api-access-9r69w\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69\" (UID: \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:08.553366 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.553328    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:08.709681 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.709648    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"]
Jul 29 10:44:08.712735 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:44:08.712690    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1d1fc7_cbfe_4393_a138_8ca52b6c853b.slice/crio-32f7f9b149c4e177ef7caf37b7267e8ec48aa054d8ce5fa2d13fd7ceccbcabfa WatchSource:0}: Error finding container 32f7f9b149c4e177ef7caf37b7267e8ec48aa054d8ce5fa2d13fd7ceccbcabfa: Status 404 returned error can't find the container with id 32f7f9b149c4e177ef7caf37b7267e8ec48aa054d8ce5fa2d13fd7ceccbcabfa
Jul 29 10:44:08.769602 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.769577    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:44:08.812002 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.811972    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa95b76-571d-45ae-8e5a-560db0dbf891-istiod-ca-cert\") pod \"caa95b76-571d-45ae-8e5a-560db0dbf891\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") "
Jul 29 10:44:08.812161 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812024    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-podinfo\") pod \"caa95b76-571d-45ae-8e5a-560db0dbf891\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") "
Jul 29 10:44:08.812161 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812047    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-socket\") pod \"caa95b76-571d-45ae-8e5a-560db0dbf891\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") "
Jul 29 10:44:08.812161 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812070    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-token\") pod \"caa95b76-571d-45ae-8e5a-560db0dbf891\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") "
Jul 29 10:44:08.812347 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812180    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6xzl\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-kube-api-access-v6xzl\") pod \"caa95b76-571d-45ae-8e5a-560db0dbf891\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") "
Jul 29 10:44:08.812347 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812277    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-envoy\") pod \"caa95b76-571d-45ae-8e5a-560db0dbf891\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") "
Jul 29 10:44:08.812347 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812313    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-credential-socket\") pod \"caa95b76-571d-45ae-8e5a-560db0dbf891\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") "
Jul 29 10:44:08.812534 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812400    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "caa95b76-571d-45ae-8e5a-560db0dbf891" (UID: "caa95b76-571d-45ae-8e5a-560db0dbf891"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:08.812534 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812492    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa95b76-571d-45ae-8e5a-560db0dbf891-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "caa95b76-571d-45ae-8e5a-560db0dbf891" (UID: "caa95b76-571d-45ae-8e5a-560db0dbf891"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:44:08.812678 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812643    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "caa95b76-571d-45ae-8e5a-560db0dbf891" (UID: "caa95b76-571d-45ae-8e5a-560db0dbf891"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:08.812763 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812657    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa95b76-571d-45ae-8e5a-560db0dbf891-istiod-ca-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:44:08.812763 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.812702    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:44:08.815051 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.815029    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "caa95b76-571d-45ae-8e5a-560db0dbf891" (UID: "caa95b76-571d-45ae-8e5a-560db0dbf891"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 10:44:08.815330 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.815261    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "caa95b76-571d-45ae-8e5a-560db0dbf891" (UID: "caa95b76-571d-45ae-8e5a-560db0dbf891"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:08.815330 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.815295    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-kube-api-access-v6xzl" (OuterVolumeSpecName: "kube-api-access-v6xzl") pod "caa95b76-571d-45ae-8e5a-560db0dbf891" (UID: "caa95b76-571d-45ae-8e5a-560db0dbf891"). InnerVolumeSpecName "kube-api-access-v6xzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:44:08.815886 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.815857    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-token" (OuterVolumeSpecName: "istio-token") pod "caa95b76-571d-45ae-8e5a-560db0dbf891" (UID: "caa95b76-571d-45ae-8e5a-560db0dbf891"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:44:08.913449 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.913381    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-data\") pod \"caa95b76-571d-45ae-8e5a-560db0dbf891\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") "
Jul 29 10:44:08.913638 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.913457    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-certs\") pod \"caa95b76-571d-45ae-8e5a-560db0dbf891\" (UID: \"caa95b76-571d-45ae-8e5a-560db0dbf891\") "
Jul 29 10:44:08.913797 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.913720    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-podinfo\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:44:08.913797 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.913740    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:44:08.913797 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.913756    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6xzl\" (UniqueName: \"kubernetes.io/projected/caa95b76-571d-45ae-8e5a-560db0dbf891-kube-api-access-v6xzl\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:44:08.913797 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.913768    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-envoy\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:44:08.913797 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.913780    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-credential-socket\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:44:08.914069 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.914012    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "caa95b76-571d-45ae-8e5a-560db0dbf891" (UID: "caa95b76-571d-45ae-8e5a-560db0dbf891"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:08.914248 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:08.914218    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-data" (OuterVolumeSpecName: "istio-data") pod "caa95b76-571d-45ae-8e5a-560db0dbf891" (UID: "caa95b76-571d-45ae-8e5a-560db0dbf891"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:09.014494 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.014456    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-istio-data\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:44:09.014494 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.014485    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa95b76-571d-45ae-8e5a-560db0dbf891-workload-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:44:09.120559 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.120524    2569 generic.go:356] "Generic (PLEG): container finished" podID="caa95b76-571d-45ae-8e5a-560db0dbf891" containerID="2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203" exitCode=0
Jul 29 10:44:09.120970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.120572    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"
Jul 29 10:44:09.120970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.120607    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9" event={"ID":"caa95b76-571d-45ae-8e5a-560db0dbf891","Type":"ContainerDied","Data":"2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203"}
Jul 29 10:44:09.120970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.120654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9" event={"ID":"caa95b76-571d-45ae-8e5a-560db0dbf891","Type":"ContainerDied","Data":"ce65ce2f18b6ed5e7e1570f3015bba5df02af5432f62a29864c0bfa28ef151fe"}
Jul 29 10:44:09.120970 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.120677    2569 scope.go:117] "RemoveContainer" containerID="2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203"
Jul 29 10:44:09.122295 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.122272    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69" event={"ID":"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b","Type":"ContainerStarted","Data":"186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e"}
Jul 29 10:44:09.122393 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.122323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69" event={"ID":"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b","Type":"ContainerStarted","Data":"32f7f9b149c4e177ef7caf37b7267e8ec48aa054d8ce5fa2d13fd7ceccbcabfa"}
Jul 29 10:44:09.122461 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.122399    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:09.130920 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.130905    2569 scope.go:117] "RemoveContainer" containerID="2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203"
Jul 29 10:44:09.131192 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:44:09.131166    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203\": container with ID starting with 2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203 not found: ID does not exist" containerID="2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203"
Jul 29 10:44:09.131282 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.131196    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203"} err="failed to get container status \"2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203\": rpc error: code = NotFound desc = could not find container \"2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203\": container with ID starting with 2e7433d0a656146d92f9e5b020e2f40b1263d888d1fb04e42a1385f06932f203 not found: ID does not exist"
Jul 29 10:44:09.142615 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.142567    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69" podStartSLOduration=1.142550571 podStartE2EDuration="1.142550571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:44:09.140529474 +0000 UTC m=+923.533742359" watchObservedRunningTime="2026-07-29 10:44:09.142550571 +0000 UTC m=+923.535763440"
Jul 29 10:44:09.154499 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.154463    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"]
Jul 29 10:44:09.160774 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:09.160748    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-7npr9"]
Jul 29 10:44:10.043240 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:10.043206    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caa95b76-571d-45ae-8e5a-560db0dbf891" path="/var/lib/kubelet/pods/caa95b76-571d-45ae-8e5a-560db0dbf891/volumes"
Jul 29 10:44:10.627508 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:10.627477    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:11.998108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:11.998073    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:12.364252 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:12.364207    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:13.650598 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:13.650569    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:18.010469 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:18.010423    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:18.075294 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:18.075245    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:19.370901 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:19.370865    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:19.470766 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:19.470733    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:29.336016 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:29.335983    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:44:40.130444 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:40.130405    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:44:43.045562 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:44:43.045527    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:00.868055 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:00.868023    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:00.962982 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:00.962951    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:07.136500 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:07.136468    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:11.592649 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.592564    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:11.653727 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.653688    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:11.749721 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.749683    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"]
Jul 29 10:45:11.750244 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.750228    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa95b76-571d-45ae-8e5a-560db0dbf891" containerName="istio-proxy"
Jul 29 10:45:11.750298 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.750248    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa95b76-571d-45ae-8e5a-560db0dbf891" containerName="istio-proxy"
Jul 29 10:45:11.750364 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.750352    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa95b76-571d-45ae-8e5a-560db0dbf891" containerName="istio-proxy"
Jul 29 10:45:11.754956 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.754935    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:45:11.757948 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.757923    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 29 10:45:11.759364 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.759339    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 29 10:45:11.759507 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.759348    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 29 10:45:11.759587 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.759380    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-dsgq6\""
Jul 29 10:45:11.759647 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.759396    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-5v7b2\""
Jul 29 10:45:11.763447 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.763402    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"]
Jul 29 10:45:11.782238 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.782212    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:11.850727 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.850651    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b357bb86-4f43-4465-88d0-95f0c17a7b57-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j\" (UID: \"b357bb86-4f43-4465-88d0-95f0c17a7b57\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:45:11.850865 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.850765    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7mq7\" (UniqueName: \"kubernetes.io/projected/b357bb86-4f43-4465-88d0-95f0c17a7b57-kube-api-access-s7mq7\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j\" (UID: \"b357bb86-4f43-4465-88d0-95f0c17a7b57\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:45:11.951409 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.951369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7mq7\" (UniqueName: \"kubernetes.io/projected/b357bb86-4f43-4465-88d0-95f0c17a7b57-kube-api-access-s7mq7\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j\" (UID: \"b357bb86-4f43-4465-88d0-95f0c17a7b57\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:45:11.951616 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.951453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b357bb86-4f43-4465-88d0-95f0c17a7b57-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j\" (UID: \"b357bb86-4f43-4465-88d0-95f0c17a7b57\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:45:11.953976 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.953956    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b357bb86-4f43-4465-88d0-95f0c17a7b57-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j\" (UID: \"b357bb86-4f43-4465-88d0-95f0c17a7b57\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:45:11.960256 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:11.960231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7mq7\" (UniqueName: \"kubernetes.io/projected/b357bb86-4f43-4465-88d0-95f0c17a7b57-kube-api-access-s7mq7\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j\" (UID: \"b357bb86-4f43-4465-88d0-95f0c17a7b57\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:45:12.067464 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:12.067413    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:45:12.201566 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:12.201545    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"]
Jul 29 10:45:12.203893 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:45:12.203865    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb357bb86_4f43_4465_88d0_95f0c17a7b57.slice/crio-f7f418638cf97f02e0993576c0246539ec18876b7be5a5ac98dbf33172db888b WatchSource:0}: Error finding container f7f418638cf97f02e0993576c0246539ec18876b7be5a5ac98dbf33172db888b: Status 404 returned error can't find the container with id f7f418638cf97f02e0993576c0246539ec18876b7be5a5ac98dbf33172db888b
Jul 29 10:45:12.420115 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:12.420018    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j" event={"ID":"b357bb86-4f43-4465-88d0-95f0c17a7b57","Type":"ContainerStarted","Data":"3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd"}
Jul 29 10:45:12.420115 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:12.420084    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j" event={"ID":"b357bb86-4f43-4465-88d0-95f0c17a7b57","Type":"ContainerStarted","Data":"f7f418638cf97f02e0993576c0246539ec18876b7be5a5ac98dbf33172db888b"}
Jul 29 10:45:12.420115 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:12.420111    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:45:12.440421 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:12.440365    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j" podStartSLOduration=1.440348255 podStartE2EDuration="1.440348255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:45:12.439473062 +0000 UTC m=+986.832685933" watchObservedRunningTime="2026-07-29 10:45:12.440348255 +0000 UTC m=+986.833561125"
Jul 29 10:45:13.362128 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:13.362097    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:13.462444 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:13.462405    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:28.429664 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:28.429627    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:39.721481 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:39.721421    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:45:43.426807 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:45:43.426775    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:47:09.878736 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:09.878708    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:24.296879 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:24.296834    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"]
Jul 29 10:47:24.308130 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:24.308090    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:24.362752 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:24.362708    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:30.130405 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:30.130359    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69" podUID="6a1d1fc7-cbfe-4393-a138-8ca52b6c853b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:47:31.090984 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:31.090944    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:31.121120 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:31.121082    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"]
Jul 29 10:47:31.168248 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:31.168215    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:33.426451 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:33.426383    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j" podUID="b357bb86-4f43-4465-88d0-95f0c17a7b57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:47:39.319258 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:39.319191    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69" podUID="6a1d1fc7-cbfe-4393-a138-8ca52b6c853b" containerName="main" containerID="cri-o://186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e" gracePeriod=45
Jul 29 10:47:39.578219 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:39.578159    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:47:39.682215 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:39.682176    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r69w\" (UniqueName: \"kubernetes.io/projected/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-kube-api-access-9r69w\") pod \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\" (UID: \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\") "
Jul 29 10:47:39.682414 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:39.682271    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-tls-certs\") pod \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\" (UID: \"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b\") "
Jul 29 10:47:39.684436 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:39.684398    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a1d1fc7-cbfe-4393-a138-8ca52b6c853b" (UID: "6a1d1fc7-cbfe-4393-a138-8ca52b6c853b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:47:39.684518 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:39.684489    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-kube-api-access-9r69w" (OuterVolumeSpecName: "kube-api-access-9r69w") pod "6a1d1fc7-cbfe-4393-a138-8ca52b6c853b" (UID: "6a1d1fc7-cbfe-4393-a138-8ca52b6c853b"). InnerVolumeSpecName "kube-api-access-9r69w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:47:39.783325 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:39.783289    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r69w\" (UniqueName: \"kubernetes.io/projected/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-kube-api-access-9r69w\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:47:39.783325 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:39.783318    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:47:40.132536 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:40.132503    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a1d1fc7-cbfe-4393-a138-8ca52b6c853b" containerID="186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e" exitCode=0
Jul 29 10:47:40.132713 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:40.132545    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"
Jul 29 10:47:40.132713 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:40.132554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69" event={"ID":"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b","Type":"ContainerDied","Data":"186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e"}
Jul 29 10:47:40.132713 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:40.132596    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69" event={"ID":"6a1d1fc7-cbfe-4393-a138-8ca52b6c853b","Type":"ContainerDied","Data":"32f7f9b149c4e177ef7caf37b7267e8ec48aa054d8ce5fa2d13fd7ceccbcabfa"}
Jul 29 10:47:40.132713 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:40.132619    2569 scope.go:117] "RemoveContainer" containerID="186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e"
Jul 29 10:47:40.141126 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:40.141104    2569 scope.go:117] "RemoveContainer" containerID="186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e"
Jul 29 10:47:40.141376 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:47:40.141356    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e\": container with ID starting with 186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e not found: ID does not exist" containerID="186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e"
Jul 29 10:47:40.141525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:40.141388    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e"} err="failed to get container status \"186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e\": rpc error: code = NotFound desc = could not find container \"186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e\": container with ID starting with 186c7df5af059c019db466b495004bb005a1f39ab1641bba02e69ed4c58d8e5e not found: ID does not exist"
Jul 29 10:47:40.150441 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:40.150402    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"]
Jul 29 10:47:40.154297 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:40.154272    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-4ck69"]
Jul 29 10:47:42.042516 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:42.042476    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1d1fc7-cbfe-4393-a138-8ca52b6c853b" path="/var/lib/kubelet/pods/6a1d1fc7-cbfe-4393-a138-8ca52b6c853b/volumes"
Jul 29 10:47:43.426910 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:43.426871    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j" podUID="b357bb86-4f43-4465-88d0-95f0c17a7b57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:47:44.225773 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:44.225736    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:44.320989 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:44.320956    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:46.141177 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:46.141140    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j" podUID="b357bb86-4f43-4465-88d0-95f0c17a7b57" containerName="main" containerID="cri-o://3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd" gracePeriod=45
Jul 29 10:47:46.389862 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:46.389835    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:47:46.441243 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:46.441214    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7mq7\" (UniqueName: \"kubernetes.io/projected/b357bb86-4f43-4465-88d0-95f0c17a7b57-kube-api-access-s7mq7\") pod \"b357bb86-4f43-4465-88d0-95f0c17a7b57\" (UID: \"b357bb86-4f43-4465-88d0-95f0c17a7b57\") "
Jul 29 10:47:46.441417 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:46.441275    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b357bb86-4f43-4465-88d0-95f0c17a7b57-tls-certs\") pod \"b357bb86-4f43-4465-88d0-95f0c17a7b57\" (UID: \"b357bb86-4f43-4465-88d0-95f0c17a7b57\") "
Jul 29 10:47:46.443422 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:46.443397    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b357bb86-4f43-4465-88d0-95f0c17a7b57-kube-api-access-s7mq7" (OuterVolumeSpecName: "kube-api-access-s7mq7") pod "b357bb86-4f43-4465-88d0-95f0c17a7b57" (UID: "b357bb86-4f43-4465-88d0-95f0c17a7b57"). InnerVolumeSpecName "kube-api-access-s7mq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:47:46.443536 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:46.443446    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b357bb86-4f43-4465-88d0-95f0c17a7b57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b357bb86-4f43-4465-88d0-95f0c17a7b57" (UID: "b357bb86-4f43-4465-88d0-95f0c17a7b57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:47:46.542623 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:46.542583    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7mq7\" (UniqueName: \"kubernetes.io/projected/b357bb86-4f43-4465-88d0-95f0c17a7b57-kube-api-access-s7mq7\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:47:46.542623 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:46.542619    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b357bb86-4f43-4465-88d0-95f0c17a7b57-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:47:47.169085 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:47.169045    2569 generic.go:356] "Generic (PLEG): container finished" podID="b357bb86-4f43-4465-88d0-95f0c17a7b57" containerID="3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd" exitCode=0
Jul 29 10:47:47.169602 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:47.169095    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"
Jul 29 10:47:47.169602 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:47.169136    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j" event={"ID":"b357bb86-4f43-4465-88d0-95f0c17a7b57","Type":"ContainerDied","Data":"3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd"}
Jul 29 10:47:47.169602 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:47.169199    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j" event={"ID":"b357bb86-4f43-4465-88d0-95f0c17a7b57","Type":"ContainerDied","Data":"f7f418638cf97f02e0993576c0246539ec18876b7be5a5ac98dbf33172db888b"}
Jul 29 10:47:47.169602 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:47.169237    2569 scope.go:117] "RemoveContainer" containerID="3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd"
Jul 29 10:47:47.177657 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:47.177641    2569 scope.go:117] "RemoveContainer" containerID="3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd"
Jul 29 10:47:47.177899 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:47:47.177882    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd\": container with ID starting with 3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd not found: ID does not exist" containerID="3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd"
Jul 29 10:47:47.177965 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:47.177906    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd"} err="failed to get container status \"3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd\": rpc error: code = NotFound desc = could not find container \"3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd\": container with ID starting with 3af8a9d0764c9ff7c26ee4c8be8f92f326ebec099f69315f57436637b66a6afd not found: ID does not exist"
Jul 29 10:47:47.192476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:47.192449    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"]
Jul 29 10:47:47.199132 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:47.199111    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schef8z8j"]
Jul 29 10:47:48.041698 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:48.041665    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b357bb86-4f43-4465-88d0-95f0c17a7b57" path="/var/lib/kubelet/pods/b357bb86-4f43-4465-88d0-95f0c17a7b57/volumes"
Jul 29 10:47:50.480055 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:50.480020    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:52.695285 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:52.695248    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:52.755927 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:52.755893    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:56.183569 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:56.183538    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:47:56.283628 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:47:56.283586    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:06.221354 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:06.221322    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:06.511923 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:06.511826    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:12.328782 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:12.328701    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:17.026523 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.026480    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:17.262920 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.262889    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"]
Jul 29 10:48:17.263257 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.263243    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b357bb86-4f43-4465-88d0-95f0c17a7b57" containerName="main"
Jul 29 10:48:17.263304 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.263258    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b357bb86-4f43-4465-88d0-95f0c17a7b57" containerName="main"
Jul 29 10:48:17.263304 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.263270    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1d1fc7-cbfe-4393-a138-8ca52b6c853b" containerName="main"
Jul 29 10:48:17.263304 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.263275    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1d1fc7-cbfe-4393-a138-8ca52b6c853b" containerName="main"
Jul 29 10:48:17.263397 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.263338    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1d1fc7-cbfe-4393-a138-8ca52b6c853b" containerName="main"
Jul 29 10:48:17.263397 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.263348    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b357bb86-4f43-4465-88d0-95f0c17a7b57" containerName="main"
Jul 29 10:48:17.266625 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.266603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:48:17.269620 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.269595    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 29 10:48:17.269753 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.269643    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-c9fcd\""
Jul 29 10:48:17.270719 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.270698    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-b44q8\""
Jul 29 10:48:17.270867 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.270698    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 29 10:48:17.270867 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.270702    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 29 10:48:17.277290 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.277237    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"]
Jul 29 10:48:17.298318 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.298294    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:17.314739 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.314712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47b71f8-dcce-40f6-8317-6224b87297a9-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw\" (UID: \"c47b71f8-dcce-40f6-8317-6224b87297a9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:48:17.314891 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.314785    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qcw\" (UniqueName: \"kubernetes.io/projected/c47b71f8-dcce-40f6-8317-6224b87297a9-kube-api-access-w9qcw\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw\" (UID: \"c47b71f8-dcce-40f6-8317-6224b87297a9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:48:17.415244 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.415209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47b71f8-dcce-40f6-8317-6224b87297a9-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw\" (UID: \"c47b71f8-dcce-40f6-8317-6224b87297a9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:48:17.415484 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.415295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qcw\" (UniqueName: \"kubernetes.io/projected/c47b71f8-dcce-40f6-8317-6224b87297a9-kube-api-access-w9qcw\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw\" (UID: \"c47b71f8-dcce-40f6-8317-6224b87297a9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:48:17.417789 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.417764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47b71f8-dcce-40f6-8317-6224b87297a9-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw\" (UID: \"c47b71f8-dcce-40f6-8317-6224b87297a9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:48:17.427144 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.427117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qcw\" (UniqueName: \"kubernetes.io/projected/c47b71f8-dcce-40f6-8317-6224b87297a9-kube-api-access-w9qcw\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw\" (UID: \"c47b71f8-dcce-40f6-8317-6224b87297a9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:48:17.580090 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.580060    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:48:17.722365 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:17.722338    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"]
Jul 29 10:48:17.724607 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:48:17.724581    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47b71f8_dcce_40f6_8317_6224b87297a9.slice/crio-776bd24cf14b1486e17f2c58fe448b3679d288d43ee5595f33e63ed3373ce1f3 WatchSource:0}: Error finding container 776bd24cf14b1486e17f2c58fe448b3679d288d43ee5595f33e63ed3373ce1f3: Status 404 returned error can't find the container with id 776bd24cf14b1486e17f2c58fe448b3679d288d43ee5595f33e63ed3373ce1f3
Jul 29 10:48:18.314794 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:18.314754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw" event={"ID":"c47b71f8-dcce-40f6-8317-6224b87297a9","Type":"ContainerStarted","Data":"d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49"}
Jul 29 10:48:18.315220 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:18.314813    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw" event={"ID":"c47b71f8-dcce-40f6-8317-6224b87297a9","Type":"ContainerStarted","Data":"776bd24cf14b1486e17f2c58fe448b3679d288d43ee5595f33e63ed3373ce1f3"}
Jul 29 10:48:18.315220 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:18.314834    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:48:18.331973 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:18.331922    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw" podStartSLOduration=1.331907596 podStartE2EDuration="1.331907596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:48:18.329976258 +0000 UTC m=+1172.723189151" watchObservedRunningTime="2026-07-29 10:48:18.331907596 +0000 UTC m=+1172.725120466"
Jul 29 10:48:18.707762 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:18.707728    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:18.807335 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:18.807301    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:28.649871 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.649834    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:28.837470 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.837422    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"]
Jul 29 10:48:28.841030 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.841008    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:48:28.844661 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.844641    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 29 10:48:28.845860 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.845841    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 29 10:48:28.845960 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.845850    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 29 10:48:28.846023 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.845860    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-65rb6\""
Jul 29 10:48:28.846023 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.845897    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-8gpj9\""
Jul 29 10:48:28.850705 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.850683    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"]
Jul 29 10:48:28.873843 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.873789    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:28.919958 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.919869    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmhjn\" (UniqueName: \"kubernetes.io/projected/a72c6e41-010f-4356-9f99-92b36561b923-kube-api-access-lmhjn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2\" (UID: \"a72c6e41-010f-4356-9f99-92b36561b923\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:48:28.919958 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:28.919909    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72c6e41-010f-4356-9f99-92b36561b923-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2\" (UID: \"a72c6e41-010f-4356-9f99-92b36561b923\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:48:29.021068 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.021038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmhjn\" (UniqueName: \"kubernetes.io/projected/a72c6e41-010f-4356-9f99-92b36561b923-kube-api-access-lmhjn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2\" (UID: \"a72c6e41-010f-4356-9f99-92b36561b923\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:48:29.021068 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.021072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72c6e41-010f-4356-9f99-92b36561b923-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2\" (UID: \"a72c6e41-010f-4356-9f99-92b36561b923\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:48:29.023725 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.023696    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72c6e41-010f-4356-9f99-92b36561b923-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2\" (UID: \"a72c6e41-010f-4356-9f99-92b36561b923\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:48:29.029880 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.029852    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmhjn\" (UniqueName: \"kubernetes.io/projected/a72c6e41-010f-4356-9f99-92b36561b923-kube-api-access-lmhjn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2\" (UID: \"a72c6e41-010f-4356-9f99-92b36561b923\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:48:29.152762 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.152724    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:48:29.295037 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.295010    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"]
Jul 29 10:48:29.297580 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:48:29.297554    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72c6e41_010f_4356_9f99_92b36561b923.slice/crio-53a88b4735f3551f975f8dbaa54b84227be1e36f2e0ee50dd88be19c6ba5abbb WatchSource:0}: Error finding container 53a88b4735f3551f975f8dbaa54b84227be1e36f2e0ee50dd88be19c6ba5abbb: Status 404 returned error can't find the container with id 53a88b4735f3551f975f8dbaa54b84227be1e36f2e0ee50dd88be19c6ba5abbb
Jul 29 10:48:29.371107 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.371077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2" event={"ID":"a72c6e41-010f-4356-9f99-92b36561b923","Type":"ContainerStarted","Data":"fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475"}
Jul 29 10:48:29.371271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.371133    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2" event={"ID":"a72c6e41-010f-4356-9f99-92b36561b923","Type":"ContainerStarted","Data":"53a88b4735f3551f975f8dbaa54b84227be1e36f2e0ee50dd88be19c6ba5abbb"}
Jul 29 10:48:29.371271 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.371210    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:48:29.389039 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:29.388983    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2" podStartSLOduration=1.388968029 podStartE2EDuration="1.388968029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:48:29.387142314 +0000 UTC m=+1183.780355222" watchObservedRunningTime="2026-07-29 10:48:29.388968029 +0000 UTC m=+1183.782180899"
Jul 29 10:48:35.121220 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:35.121181    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:35.177235 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:35.177202    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:48.835594 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:48.835541    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:48:49.322715 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:48:49.322638    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:49:00.378420 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:00.378388    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:49:01.555322 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:01.555286    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"]
Jul 29 10:49:01.561050 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:01.560989    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:01.627306 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:01.627268    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:10.377878 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:10.377841    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2" podUID="a72c6e41-010f-4356-9f99-92b36561b923" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:49:16.575980 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:16.575926    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2" podUID="a72c6e41-010f-4356-9f99-92b36561b923" containerName="main" containerID="cri-o://fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475" gracePeriod=45
Jul 29 10:49:16.838596 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:16.838567    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:49:16.951220 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:16.951193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72c6e41-010f-4356-9f99-92b36561b923-tls-certs\") pod \"a72c6e41-010f-4356-9f99-92b36561b923\" (UID: \"a72c6e41-010f-4356-9f99-92b36561b923\") "
Jul 29 10:49:16.951455 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:16.951267    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmhjn\" (UniqueName: \"kubernetes.io/projected/a72c6e41-010f-4356-9f99-92b36561b923-kube-api-access-lmhjn\") pod \"a72c6e41-010f-4356-9f99-92b36561b923\" (UID: \"a72c6e41-010f-4356-9f99-92b36561b923\") "
Jul 29 10:49:16.953464 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:16.953410    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72c6e41-010f-4356-9f99-92b36561b923-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a72c6e41-010f-4356-9f99-92b36561b923" (UID: "a72c6e41-010f-4356-9f99-92b36561b923"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:49:16.953582 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:16.953486    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72c6e41-010f-4356-9f99-92b36561b923-kube-api-access-lmhjn" (OuterVolumeSpecName: "kube-api-access-lmhjn") pod "a72c6e41-010f-4356-9f99-92b36561b923" (UID: "a72c6e41-010f-4356-9f99-92b36561b923"). InnerVolumeSpecName "kube-api-access-lmhjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:49:17.051840 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.051803    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmhjn\" (UniqueName: \"kubernetes.io/projected/a72c6e41-010f-4356-9f99-92b36561b923-kube-api-access-lmhjn\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:49:17.051840 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.051832    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72c6e41-010f-4356-9f99-92b36561b923-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:49:17.606090 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.606053    2569 generic.go:356] "Generic (PLEG): container finished" podID="a72c6e41-010f-4356-9f99-92b36561b923" containerID="fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475" exitCode=0
Jul 29 10:49:17.606528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.606109    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"
Jul 29 10:49:17.606528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.606149    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2" event={"ID":"a72c6e41-010f-4356-9f99-92b36561b923","Type":"ContainerDied","Data":"fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475"}
Jul 29 10:49:17.606528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.606223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2" event={"ID":"a72c6e41-010f-4356-9f99-92b36561b923","Type":"ContainerDied","Data":"53a88b4735f3551f975f8dbaa54b84227be1e36f2e0ee50dd88be19c6ba5abbb"}
Jul 29 10:49:17.606528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.606256    2569 scope.go:117] "RemoveContainer" containerID="fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475"
Jul 29 10:49:17.615156 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.615140    2569 scope.go:117] "RemoveContainer" containerID="fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475"
Jul 29 10:49:17.615402 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:49:17.615384    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475\": container with ID starting with fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475 not found: ID does not exist" containerID="fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475"
Jul 29 10:49:17.615508 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.615409    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475"} err="failed to get container status \"fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475\": rpc error: code = NotFound desc = could not find container \"fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475\": container with ID starting with fb4016d5d23ffb00cc374316099c0f9d4b5c9aa4d634672cab7f9320997d0475 not found: ID does not exist"
Jul 29 10:49:17.635577 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.635544    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"]
Jul 29 10:49:17.637929 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:17.637903    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5t4qf2"]
Jul 29 10:49:18.042233 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:18.042153    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a72c6e41-010f-4356-9f99-92b36561b923" path="/var/lib/kubelet/pods/a72c6e41-010f-4356-9f99-92b36561b923/volumes"
Jul 29 10:49:21.477052 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:21.477018    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:21.577042 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:21.577008    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:23.961246 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:23.961211    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:24.142701 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.142667    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"]
Jul 29 10:49:24.143045 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.143032    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a72c6e41-010f-4356-9f99-92b36561b923" containerName="main"
Jul 29 10:49:24.143045 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.143046    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a72c6e41-010f-4356-9f99-92b36561b923" containerName="main"
Jul 29 10:49:24.143166 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.143101    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a72c6e41-010f-4356-9f99-92b36561b923" containerName="main"
Jul 29 10:49:24.147696 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.147671    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:24.151707 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.151674    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-7cjvh\""
Jul 29 10:49:24.151912 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.151739    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 29 10:49:24.151912 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.151674    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 29 10:49:24.152014 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.151935    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 29 10:49:24.152014 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.151942    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-8gd7q\""
Jul 29 10:49:24.159902 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.159882    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"]
Jul 29 10:49:24.183735 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.183701    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:24.211902 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.211827    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxd42\" (UniqueName: \"kubernetes.io/projected/84d4cc4e-f4d3-4386-96aa-6342cedc786c-kube-api-access-nxd42\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn\" (UID: \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:24.211902 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.211870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d4cc4e-f4d3-4386-96aa-6342cedc786c-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn\" (UID: \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:24.313355 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.313323    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxd42\" (UniqueName: \"kubernetes.io/projected/84d4cc4e-f4d3-4386-96aa-6342cedc786c-kube-api-access-nxd42\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn\" (UID: \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:24.313561 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.313369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d4cc4e-f4d3-4386-96aa-6342cedc786c-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn\" (UID: \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:24.315923 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.315902    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d4cc4e-f4d3-4386-96aa-6342cedc786c-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn\" (UID: \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:24.321241 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.321216    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxd42\" (UniqueName: \"kubernetes.io/projected/84d4cc4e-f4d3-4386-96aa-6342cedc786c-kube-api-access-nxd42\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn\" (UID: \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:24.459267 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.459232    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:24.599444 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.599365    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"]
Jul 29 10:49:24.602538 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:49:24.602510    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d4cc4e_f4d3_4386_96aa_6342cedc786c.slice/crio-1a40b8aa212ff8747f5e40216c75f7e8687eada72145a40c16da75a026f39d49 WatchSource:0}: Error finding container 1a40b8aa212ff8747f5e40216c75f7e8687eada72145a40c16da75a026f39d49: Status 404 returned error can't find the container with id 1a40b8aa212ff8747f5e40216c75f7e8687eada72145a40c16da75a026f39d49
Jul 29 10:49:24.643352 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:24.643323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn" event={"ID":"84d4cc4e-f4d3-4386-96aa-6342cedc786c","Type":"ContainerStarted","Data":"1a40b8aa212ff8747f5e40216c75f7e8687eada72145a40c16da75a026f39d49"}
Jul 29 10:49:25.649723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:25.649691    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn" event={"ID":"84d4cc4e-f4d3-4386-96aa-6342cedc786c","Type":"ContainerStarted","Data":"39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c"}
Jul 29 10:49:25.650099 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:25.649815    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:25.666678 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:25.666624    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn" podStartSLOduration=1.6666077769999998 podStartE2EDuration="1.666607777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:49:25.665108441 +0000 UTC m=+1240.058321312" watchObservedRunningTime="2026-07-29 10:49:25.666607777 +0000 UTC m=+1240.059820648"
Jul 29 10:49:28.191554 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:28.191521    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:30.646393 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:30.646340    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:33.773459 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:33.773401    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:33.871215 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:33.871180    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:56.656932 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:56.656904    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:49:58.050171 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:58.050139    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"]
Jul 29 10:49:58.076905 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:58.076854    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:49:58.230795 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:49:58.230765    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:06.656481 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:06.656422    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn" podUID="84d4cc4e-f4d3-4386-96aa-6342cedc786c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:50:13.070090 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.070041    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn" podUID="84d4cc4e-f4d3-4386-96aa-6342cedc786c" containerName="main" containerID="cri-o://39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c" gracePeriod=45
Jul 29 10:50:13.321999 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.321939    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:50:13.456003 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.455970    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d4cc4e-f4d3-4386-96aa-6342cedc786c-tls-certs\") pod \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\" (UID: \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\") "
Jul 29 10:50:13.456184 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.456107    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxd42\" (UniqueName: \"kubernetes.io/projected/84d4cc4e-f4d3-4386-96aa-6342cedc786c-kube-api-access-nxd42\") pod \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\" (UID: \"84d4cc4e-f4d3-4386-96aa-6342cedc786c\") "
Jul 29 10:50:13.458193 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.458167    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d4cc4e-f4d3-4386-96aa-6342cedc786c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84d4cc4e-f4d3-4386-96aa-6342cedc786c" (UID: "84d4cc4e-f4d3-4386-96aa-6342cedc786c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:50:13.458315 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.458278    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d4cc4e-f4d3-4386-96aa-6342cedc786c-kube-api-access-nxd42" (OuterVolumeSpecName: "kube-api-access-nxd42") pod "84d4cc4e-f4d3-4386-96aa-6342cedc786c" (UID: "84d4cc4e-f4d3-4386-96aa-6342cedc786c"). InnerVolumeSpecName "kube-api-access-nxd42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:50:13.557407 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.557370    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxd42\" (UniqueName: \"kubernetes.io/projected/84d4cc4e-f4d3-4386-96aa-6342cedc786c-kube-api-access-nxd42\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:50:13.557407 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.557398    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d4cc4e-f4d3-4386-96aa-6342cedc786c-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:50:13.888366 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.888334    2569 generic.go:356] "Generic (PLEG): container finished" podID="84d4cc4e-f4d3-4386-96aa-6342cedc786c" containerID="39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c" exitCode=0
Jul 29 10:50:13.888578 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.888377    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"
Jul 29 10:50:13.888578 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.888412    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn" event={"ID":"84d4cc4e-f4d3-4386-96aa-6342cedc786c","Type":"ContainerDied","Data":"39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c"}
Jul 29 10:50:13.888578 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.888487    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn" event={"ID":"84d4cc4e-f4d3-4386-96aa-6342cedc786c","Type":"ContainerDied","Data":"1a40b8aa212ff8747f5e40216c75f7e8687eada72145a40c16da75a026f39d49"}
Jul 29 10:50:13.888578 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.888519    2569 scope.go:117] "RemoveContainer" containerID="39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c"
Jul 29 10:50:13.897254 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.897238    2569 scope.go:117] "RemoveContainer" containerID="39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c"
Jul 29 10:50:13.897516 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:50:13.897498    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c\": container with ID starting with 39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c not found: ID does not exist" containerID="39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c"
Jul 29 10:50:13.897577 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.897526    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c"} err="failed to get container status \"39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c\": rpc error: code = NotFound desc = could not find container \"39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c\": container with ID starting with 39df76c0d218cdc20730b69c7e3da64023cb99574bc36e471ac567c616926e5c not found: ID does not exist"
Jul 29 10:50:13.910576 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.910554    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"]
Jul 29 10:50:13.914230 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:13.914207    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schesmlpn"]
Jul 29 10:50:14.042287 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:14.042257    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d4cc4e-f4d3-4386-96aa-6342cedc786c" path="/var/lib/kubelet/pods/84d4cc4e-f4d3-4386-96aa-6342cedc786c/volumes"
Jul 29 10:50:18.014039 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:18.014006    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:18.110571 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:18.110539    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:23.801535 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:23.801501    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:26.153228 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.153200    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:26.330029 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.329994    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"]
Jul 29 10:50:26.330582 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.330561    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d4cc4e-f4d3-4386-96aa-6342cedc786c" containerName="main"
Jul 29 10:50:26.330659 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.330585    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d4cc4e-f4d3-4386-96aa-6342cedc786c" containerName="main"
Jul 29 10:50:26.330716 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.330705    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d4cc4e-f4d3-4386-96aa-6342cedc786c" containerName="main"
Jul 29 10:50:26.333676 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.333659    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:50:26.337717 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.337697    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 29 10:50:26.337881 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.337803    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-8jtr7\""
Jul 29 10:50:26.337881 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.337813    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 29 10:50:26.337881 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.337836    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 29 10:50:26.338043 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.337897    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-4dcrk\""
Jul 29 10:50:26.342342 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.342319    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"]
Jul 29 10:50:26.350952 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.350922    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:26.476834 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.476745    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cb090c-99eb-4216-be1e-9bce30c5933f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm\" (UID: \"07cb090c-99eb-4216-be1e-9bce30c5933f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:50:26.476834 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.476786    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj77r\" (UniqueName: \"kubernetes.io/projected/07cb090c-99eb-4216-be1e-9bce30c5933f-kube-api-access-hj77r\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm\" (UID: \"07cb090c-99eb-4216-be1e-9bce30c5933f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:50:26.577907 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.577872    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cb090c-99eb-4216-be1e-9bce30c5933f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm\" (UID: \"07cb090c-99eb-4216-be1e-9bce30c5933f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:50:26.578086 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.577910    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj77r\" (UniqueName: \"kubernetes.io/projected/07cb090c-99eb-4216-be1e-9bce30c5933f-kube-api-access-hj77r\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm\" (UID: \"07cb090c-99eb-4216-be1e-9bce30c5933f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:50:26.580741 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.580709    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cb090c-99eb-4216-be1e-9bce30c5933f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm\" (UID: \"07cb090c-99eb-4216-be1e-9bce30c5933f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:50:26.587107 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.587075    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj77r\" (UniqueName: \"kubernetes.io/projected/07cb090c-99eb-4216-be1e-9bce30c5933f-kube-api-access-hj77r\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm\" (UID: \"07cb090c-99eb-4216-be1e-9bce30c5933f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:50:26.645108 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.645045    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:50:26.789453 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.789408    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"]
Jul 29 10:50:26.791420 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:50:26.791380    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cb090c_99eb_4216_be1e_9bce30c5933f.slice/crio-56a18570aee6eae5c5bdda29e07df535e3e4de7e156bc064184f5e39a9972c2b WatchSource:0}: Error finding container 56a18570aee6eae5c5bdda29e07df535e3e4de7e156bc064184f5e39a9972c2b: Status 404 returned error can't find the container with id 56a18570aee6eae5c5bdda29e07df535e3e4de7e156bc064184f5e39a9972c2b
Jul 29 10:50:26.952672 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.952631    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm" event={"ID":"07cb090c-99eb-4216-be1e-9bce30c5933f","Type":"ContainerStarted","Data":"9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a"}
Jul 29 10:50:26.952866 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.952695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm" event={"ID":"07cb090c-99eb-4216-be1e-9bce30c5933f","Type":"ContainerStarted","Data":"56a18570aee6eae5c5bdda29e07df535e3e4de7e156bc064184f5e39a9972c2b"}
Jul 29 10:50:26.952866 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.952722    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:50:26.972524 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:26.972479    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm" podStartSLOduration=0.97246282 podStartE2EDuration="972.46282ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:50:26.971080695 +0000 UTC m=+1301.364293559" watchObservedRunningTime="2026-07-29 10:50:26.97246282 +0000 UTC m=+1301.365675690"
Jul 29 10:50:29.132775 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:29.132735    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"]
Jul 29 10:50:29.135603 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:29.135537    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:29.198091 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:29.198056    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:29.322944 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:29.322907    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw" podUID="c47b71f8-dcce-40f6-8317-6224b87297a9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:50:29.999583 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:29.999550    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:30.059233 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:30.059200    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:38.545834 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:38.545783    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:39.323178 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:39.323135    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw" podUID="c47b71f8-dcce-40f6-8317-6224b87297a9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:50:44.155319 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:44.155281    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw" podUID="c47b71f8-dcce-40f6-8317-6224b87297a9" containerName="main" containerID="cri-o://d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49" gracePeriod=45
Jul 29 10:50:44.408301 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:44.408238    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:50:44.433840 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:44.433806    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9qcw\" (UniqueName: \"kubernetes.io/projected/c47b71f8-dcce-40f6-8317-6224b87297a9-kube-api-access-w9qcw\") pod \"c47b71f8-dcce-40f6-8317-6224b87297a9\" (UID: \"c47b71f8-dcce-40f6-8317-6224b87297a9\") "
Jul 29 10:50:44.433997 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:44.433958    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47b71f8-dcce-40f6-8317-6224b87297a9-tls-certs\") pod \"c47b71f8-dcce-40f6-8317-6224b87297a9\" (UID: \"c47b71f8-dcce-40f6-8317-6224b87297a9\") "
Jul 29 10:50:44.435962 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:44.435931    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47b71f8-dcce-40f6-8317-6224b87297a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c47b71f8-dcce-40f6-8317-6224b87297a9" (UID: "c47b71f8-dcce-40f6-8317-6224b87297a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:50:44.436082 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:44.435996    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47b71f8-dcce-40f6-8317-6224b87297a9-kube-api-access-w9qcw" (OuterVolumeSpecName: "kube-api-access-w9qcw") pod "c47b71f8-dcce-40f6-8317-6224b87297a9" (UID: "c47b71f8-dcce-40f6-8317-6224b87297a9"). InnerVolumeSpecName "kube-api-access-w9qcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:50:44.534617 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:44.534577    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47b71f8-dcce-40f6-8317-6224b87297a9-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:50:44.534617 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:44.534612    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9qcw\" (UniqueName: \"kubernetes.io/projected/c47b71f8-dcce-40f6-8317-6224b87297a9-kube-api-access-w9qcw\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:50:45.042174 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:45.042140    2569 generic.go:356] "Generic (PLEG): container finished" podID="c47b71f8-dcce-40f6-8317-6224b87297a9" containerID="d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49" exitCode=0
Jul 29 10:50:45.042389 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:45.042188    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"
Jul 29 10:50:45.042389 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:45.042223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw" event={"ID":"c47b71f8-dcce-40f6-8317-6224b87297a9","Type":"ContainerDied","Data":"d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49"}
Jul 29 10:50:45.042389 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:45.042279    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw" event={"ID":"c47b71f8-dcce-40f6-8317-6224b87297a9","Type":"ContainerDied","Data":"776bd24cf14b1486e17f2c58fe448b3679d288d43ee5595f33e63ed3373ce1f3"}
Jul 29 10:50:45.042389 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:45.042302    2569 scope.go:117] "RemoveContainer" containerID="d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49"
Jul 29 10:50:45.051973 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:45.051954    2569 scope.go:117] "RemoveContainer" containerID="d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49"
Jul 29 10:50:45.052236 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:50:45.052215    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49\": container with ID starting with d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49 not found: ID does not exist" containerID="d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49"
Jul 29 10:50:45.052302 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:45.052243    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49"} err="failed to get container status \"d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49\": rpc error: code = NotFound desc = could not find container \"d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49\": container with ID starting with d2b23ed645779beedf99ea0efc51b2744d58893f4f0e9d570d99c75c1f8a8d49 not found: ID does not exist"
Jul 29 10:50:45.065699 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:45.065671    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"]
Jul 29 10:50:45.072045 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:45.072022    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxmqw"]
Jul 29 10:50:46.042186 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:46.042157    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c47b71f8-dcce-40f6-8317-6224b87297a9" path="/var/lib/kubelet/pods/c47b71f8-dcce-40f6-8317-6224b87297a9/volumes"
Jul 29 10:50:49.623443 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:49.623409    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:49.711761 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:49.711729    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:55.655255 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:55.655224    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:50:57.959265 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:50:57.959234    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:51:01.430831 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:01.430783    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:01.531961 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:01.531925    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:02.569570 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.569542    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:02.732943 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.732902    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"]
Jul 29 10:51:02.733289 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.733272    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47b71f8-dcce-40f6-8317-6224b87297a9" containerName="main"
Jul 29 10:51:02.733289 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.733287    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47b71f8-dcce-40f6-8317-6224b87297a9" containerName="main"
Jul 29 10:51:02.733451 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.733348    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47b71f8-dcce-40f6-8317-6224b87297a9" containerName="main"
Jul 29 10:51:02.738007 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.737981    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:02.740752 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.740726    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 29 10:51:02.742070 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.742039    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 29 10:51:02.742191 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.742097    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-s526k\""
Jul 29 10:51:02.742191 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.742044    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 29 10:51:02.742323 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.742230    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-hqq7f\""
Jul 29 10:51:02.750367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.750343    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"]
Jul 29 10:51:02.769831 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.769805    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:02.789017 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.788987    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22\" (UID: \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:02.789153 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.789065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ts8h\" (UniqueName: \"kubernetes.io/projected/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-kube-api-access-4ts8h\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22\" (UID: \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:02.890183 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.890152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22\" (UID: \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:02.890369 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.890215    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ts8h\" (UniqueName: \"kubernetes.io/projected/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-kube-api-access-4ts8h\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22\" (UID: \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:02.892858 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.892827    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22\" (UID: \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:02.902131 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:02.902106    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ts8h\" (UniqueName: \"kubernetes.io/projected/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-kube-api-access-4ts8h\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22\" (UID: \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:03.052210 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:03.052178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:03.187756 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:03.187714    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"]
Jul 29 10:51:03.189917 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:51:03.189892    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c42e08_fd1c_4f02_93cd_a0234bbb3ea7.slice/crio-f680c376c38786d74a8cbe068080f58c0d6bd9e27f89fe8eb31542361856d25f WatchSource:0}: Error finding container f680c376c38786d74a8cbe068080f58c0d6bd9e27f89fe8eb31542361856d25f: Status 404 returned error can't find the container with id f680c376c38786d74a8cbe068080f58c0d6bd9e27f89fe8eb31542361856d25f
Jul 29 10:51:04.132925 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:04.132889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22" event={"ID":"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7","Type":"ContainerStarted","Data":"4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684"}
Jul 29 10:51:04.133340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:04.132943    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22" event={"ID":"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7","Type":"ContainerStarted","Data":"f680c376c38786d74a8cbe068080f58c0d6bd9e27f89fe8eb31542361856d25f"}
Jul 29 10:51:04.133340 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:04.133033    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:04.152197 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:04.152147    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22" podStartSLOduration=2.152127322 podStartE2EDuration="2.152127322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:51:04.15150889 +0000 UTC m=+1338.544721761" watchObservedRunningTime="2026-07-29 10:51:04.152127322 +0000 UTC m=+1338.545340192"
Jul 29 10:51:07.689403 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:07.689370    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:07.787070 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:07.787039    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:10.579412 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:10.579375    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:11.931183 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:11.931147    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:11.932571 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:11.932546    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"]
Jul 29 10:51:12.027961 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:12.027933    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:17.959645 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:17.959605    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm" podUID="07cb090c-99eb-4216-be1e-9bce30c5933f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:51:26.952367 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:26.952326    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm" podUID="07cb090c-99eb-4216-be1e-9bce30c5933f" containerName="main" containerID="cri-o://9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a" gracePeriod=45
Jul 29 10:51:27.210749 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.210684    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:51:27.248538 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.248504    2569 generic.go:356] "Generic (PLEG): container finished" podID="07cb090c-99eb-4216-be1e-9bce30c5933f" containerID="9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a" exitCode=0
Jul 29 10:51:27.248703 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.248552    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"
Jul 29 10:51:27.248703 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.248578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm" event={"ID":"07cb090c-99eb-4216-be1e-9bce30c5933f","Type":"ContainerDied","Data":"9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a"}
Jul 29 10:51:27.248703 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.248630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm" event={"ID":"07cb090c-99eb-4216-be1e-9bce30c5933f","Type":"ContainerDied","Data":"56a18570aee6eae5c5bdda29e07df535e3e4de7e156bc064184f5e39a9972c2b"}
Jul 29 10:51:27.248703 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.248661    2569 scope.go:117] "RemoveContainer" containerID="9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a"
Jul 29 10:51:27.257489 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.257470    2569 scope.go:117] "RemoveContainer" containerID="9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a"
Jul 29 10:51:27.257765 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:51:27.257746    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a\": container with ID starting with 9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a not found: ID does not exist" containerID="9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a"
Jul 29 10:51:27.257867 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.257773    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a"} err="failed to get container status \"9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a\": rpc error: code = NotFound desc = could not find container \"9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a\": container with ID starting with 9e261c2bcd260977ffd3ed65721d2cea68bed10c394410ff3b6dc4200cb1447a not found: ID does not exist"
Jul 29 10:51:27.314691 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.314657    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cb090c-99eb-4216-be1e-9bce30c5933f-tls-certs\") pod \"07cb090c-99eb-4216-be1e-9bce30c5933f\" (UID: \"07cb090c-99eb-4216-be1e-9bce30c5933f\") "
Jul 29 10:51:27.314858 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.314706    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj77r\" (UniqueName: \"kubernetes.io/projected/07cb090c-99eb-4216-be1e-9bce30c5933f-kube-api-access-hj77r\") pod \"07cb090c-99eb-4216-be1e-9bce30c5933f\" (UID: \"07cb090c-99eb-4216-be1e-9bce30c5933f\") "
Jul 29 10:51:27.316927 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.316900    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07cb090c-99eb-4216-be1e-9bce30c5933f-kube-api-access-hj77r" (OuterVolumeSpecName: "kube-api-access-hj77r") pod "07cb090c-99eb-4216-be1e-9bce30c5933f" (UID: "07cb090c-99eb-4216-be1e-9bce30c5933f"). InnerVolumeSpecName "kube-api-access-hj77r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:51:27.316927 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.316909    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07cb090c-99eb-4216-be1e-9bce30c5933f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07cb090c-99eb-4216-be1e-9bce30c5933f" (UID: "07cb090c-99eb-4216-be1e-9bce30c5933f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:51:27.415998 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.415958    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cb090c-99eb-4216-be1e-9bce30c5933f-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:51:27.415998 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.415988    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj77r\" (UniqueName: \"kubernetes.io/projected/07cb090c-99eb-4216-be1e-9bce30c5933f-kube-api-access-hj77r\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:51:27.577773 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.577738    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"]
Jul 29 10:51:27.582153 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:27.582124    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-565876zfjm"]
Jul 29 10:51:28.042343 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:28.042313    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07cb090c-99eb-4216-be1e-9bce30c5933f" path="/var/lib/kubelet/pods/07cb090c-99eb-4216-be1e-9bce30c5933f/volumes"
Jul 29 10:51:32.276509 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:32.276465    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:32.382774 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:32.382736    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:35.139657 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:35.139627    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:51:36.461333 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:36.461303    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:36.668899 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:36.668862    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:39.453734 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:39.453697    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:44.936628 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:44.936592    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:45.006318 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:45.006288    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:56.031542 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:56.031507    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:51:58.534443 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:51:58.534397    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:12.474476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:12.474442    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:12.535874 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:12.535834    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:32.575593 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:32.575556    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:32.673114 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:32.673078    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:39.143351 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:39.143316    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:39.621182 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:39.621148    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:39.815949 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:39.815913    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:44.923758 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:44.923720    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:45.028834 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:45.028800    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:52:46.228613 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:52:46.228575    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:06.922476 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:06.922421    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"]
Jul 29 10:53:06.930323 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:06.930285    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:06.996612 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:06.996581    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:12.764482 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:12.764444    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:12.832733 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:12.832699    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:15.140054 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:15.139991    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22" podUID="b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:53:21.943565 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:21.943496    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22" podUID="b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7" containerName="main" containerID="cri-o://4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684" gracePeriod=45
Jul 29 10:53:22.206046 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.205990    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:53:22.238785 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.238752    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ts8h\" (UniqueName: \"kubernetes.io/projected/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-kube-api-access-4ts8h\") pod \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\" (UID: \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\") "
Jul 29 10:53:22.238968 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.238836    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-tls-certs\") pod \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\" (UID: \"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7\") "
Jul 29 10:53:22.241153 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.241121    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-kube-api-access-4ts8h" (OuterVolumeSpecName: "kube-api-access-4ts8h") pod "b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7" (UID: "b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7"). InnerVolumeSpecName "kube-api-access-4ts8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:53:22.241272 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.241218    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7" (UID: "b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:53:22.339525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.339489    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:53:22.339525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.339521    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ts8h\" (UniqueName: \"kubernetes.io/projected/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7-kube-api-access-4ts8h\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:53:22.811754 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.811720    2569 generic.go:356] "Generic (PLEG): container finished" podID="b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7" containerID="4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684" exitCode=0
Jul 29 10:53:22.811939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.811766    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"
Jul 29 10:53:22.811939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.811802    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22" event={"ID":"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7","Type":"ContainerDied","Data":"4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684"}
Jul 29 10:53:22.811939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.811856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22" event={"ID":"b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7","Type":"ContainerDied","Data":"f680c376c38786d74a8cbe068080f58c0d6bd9e27f89fe8eb31542361856d25f"}
Jul 29 10:53:22.811939 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.811879    2569 scope.go:117] "RemoveContainer" containerID="4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684"
Jul 29 10:53:22.821252 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.821234    2569 scope.go:117] "RemoveContainer" containerID="4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684"
Jul 29 10:53:22.821555 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:53:22.821533    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684\": container with ID starting with 4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684 not found: ID does not exist" containerID="4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684"
Jul 29 10:53:22.821614 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.821563    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684"} err="failed to get container status \"4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684\": rpc error: code = NotFound desc = could not find container \"4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684\": container with ID starting with 4736a432b4c49432272c0f5470d10a2dd604dadd80835d1894c99bbfbdf02684 not found: ID does not exist"
Jul 29 10:53:22.834484 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.834461    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"]
Jul 29 10:53:22.837986 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:22.837963    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-6gc22"]
Jul 29 10:53:24.041696 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:24.041660    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7" path="/var/lib/kubelet/pods/b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7/volumes"
Jul 29 10:53:26.994316 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:26.994282    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:27.086789 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:27.086756    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:31.158163 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.158132    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:31.364047 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.364015    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"]
Jul 29 10:53:31.364384 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.364372    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07cb090c-99eb-4216-be1e-9bce30c5933f" containerName="main"
Jul 29 10:53:31.364444 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.364386    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="07cb090c-99eb-4216-be1e-9bce30c5933f" containerName="main"
Jul 29 10:53:31.364444 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.364405    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7" containerName="main"
Jul 29 10:53:31.364444 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.364410    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7" containerName="main"
Jul 29 10:53:31.364558 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.364489    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="07cb090c-99eb-4216-be1e-9bce30c5933f" containerName="main"
Jul 29 10:53:31.364558 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.364500    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c42e08-fd1c-4f02-93cd-a0234bbb3ea7" containerName="main"
Jul 29 10:53:31.369110 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.369088    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:53:31.372221 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.372195    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 29 10:53:31.373337 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.373317    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-epp-sa-dockercfg-b685n\""
Jul 29 10:53:31.373715 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.373690    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-h5sj8\""
Jul 29 10:53:31.373715 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.373708    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 29 10:53:31.373877 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.373707    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 29 10:53:31.382231 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.382207    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"]
Jul 29 10:53:31.404653 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.404627    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:31.415827 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.415757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971bc44f-98be-4b97-801f-cbabc9c334fe-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n\" (UID: \"971bc44f-98be-4b97-801f-cbabc9c334fe\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:53:31.415827 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.415821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/971bc44f-98be-4b97-801f-cbabc9c334fe-kube-api-access-vmgwz\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n\" (UID: \"971bc44f-98be-4b97-801f-cbabc9c334fe\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:53:31.516308 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.516274    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971bc44f-98be-4b97-801f-cbabc9c334fe-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n\" (UID: \"971bc44f-98be-4b97-801f-cbabc9c334fe\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:53:31.516557 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.516482    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/971bc44f-98be-4b97-801f-cbabc9c334fe-kube-api-access-vmgwz\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n\" (UID: \"971bc44f-98be-4b97-801f-cbabc9c334fe\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:53:31.518918 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.518893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971bc44f-98be-4b97-801f-cbabc9c334fe-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n\" (UID: \"971bc44f-98be-4b97-801f-cbabc9c334fe\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:53:31.525611 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.525584    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/971bc44f-98be-4b97-801f-cbabc9c334fe-kube-api-access-vmgwz\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n\" (UID: \"971bc44f-98be-4b97-801f-cbabc9c334fe\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:53:31.682115 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.682020    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:53:31.827508 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.827384    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"]
Jul 29 10:53:31.829862 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:53:31.829837    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971bc44f_98be_4b97_801f_cbabc9c334fe.slice/crio-1be0f958ddf557991f0f51493cf329bdd8e18f0854043c540275fdc758de2b5d WatchSource:0}: Error finding container 1be0f958ddf557991f0f51493cf329bdd8e18f0854043c540275fdc758de2b5d: Status 404 returned error can't find the container with id 1be0f958ddf557991f0f51493cf329bdd8e18f0854043c540275fdc758de2b5d
Jul 29 10:53:31.863336 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:31.863302    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n" event={"ID":"971bc44f-98be-4b97-801f-cbabc9c334fe","Type":"ContainerStarted","Data":"1be0f958ddf557991f0f51493cf329bdd8e18f0854043c540275fdc758de2b5d"}
Jul 29 10:53:32.479568 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:32.479538    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:32.578361 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:32.578310    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:32.869830 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:32.869796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n" event={"ID":"971bc44f-98be-4b97-801f-cbabc9c334fe","Type":"ContainerStarted","Data":"1c28763794c1ab583920e367774678ee3607ec79d8dbd92be025450c97f8cd46"}
Jul 29 10:53:32.869978 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:32.869891    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:53:32.890520 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:32.890460    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n" podStartSLOduration=1.89044217 podStartE2EDuration="1.89044217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:53:32.888647113 +0000 UTC m=+1487.281859995" watchObservedRunningTime="2026-07-29 10:53:32.89044217 +0000 UTC m=+1487.283655034"
Jul 29 10:53:34.608963 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:34.608919    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:38.224883 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:38.224845    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:38.547799 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:38.547711    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:38.755629 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:38.755590    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:40.338252 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:40.338222    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:40.489645 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:40.489611    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:44.246525 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:44.246487    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:53:44.316106 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:53:44.316073    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:03.094563 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.094525    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"]
Jul 29 10:54:03.101268 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.101235    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.104391 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.104365    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 29 10:54:03.105652 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.105618    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 29 10:54:03.105845 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.105630    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 29 10:54:03.105845 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.105692    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-87z5c\""
Jul 29 10:54:03.108887 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.108861    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"]
Jul 29 10:54:03.205450 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.205400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.205622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.205498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.205622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.205547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.205622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.205573    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.205622 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.205592    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67ffg\" (UniqueName: \"kubernetes.io/projected/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-kube-api-access-67ffg\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.205825 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.205706    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.307058 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.307019    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.307215 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.307086    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.307215 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.307123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.307215 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.307159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.307215 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.307186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67ffg\" (UniqueName: \"kubernetes.io/projected/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-kube-api-access-67ffg\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.307473 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.307232    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.307564 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.307538    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.307633 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.307598    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.307739 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.307718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.309529 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.309509    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.309700 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.309685    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.315337 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.315311    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67ffg\" (UniqueName: \"kubernetes.io/projected/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-kube-api-access-67ffg\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.414219 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.414129    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:03.554512 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.554416    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"]
Jul 29 10:54:03.556963 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:54:03.556938    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f6b58c_f9cc_4b09_8510_9ab40d91ef25.slice/crio-fde9c60ccb7f3e8eb73278482cd5fbccc612a2245ec1501055fbdc80f9aabb7b WatchSource:0}: Error finding container fde9c60ccb7f3e8eb73278482cd5fbccc612a2245ec1501055fbdc80f9aabb7b: Status 404 returned error can't find the container with id fde9c60ccb7f3e8eb73278482cd5fbccc612a2245ec1501055fbdc80f9aabb7b
Jul 29 10:54:03.558738 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.558722    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:54:03.876753 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.876725    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 10:54:03.953785 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:03.953737    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:04.013259 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:04.013225    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp" event={"ID":"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25","Type":"ContainerStarted","Data":"fde9c60ccb7f3e8eb73278482cd5fbccc612a2245ec1501055fbdc80f9aabb7b"}
Jul 29 10:54:04.050454 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:04.050408    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:06.025280 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:06.025239    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp" event={"ID":"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25","Type":"ContainerStarted","Data":"209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086"}
Jul 29 10:54:06.044283 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:06.044233    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp" podStartSLOduration=1.39876105 podStartE2EDuration="3.044217128s" totalImagesPullingTime="1.645456078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:54:06.041675565 +0000 UTC m=+1520.434888462" watchObservedRunningTime="2026-07-29 10:54:06.044217128 +0000 UTC m=+1520.437429998"
Jul 29 10:54:06.414518 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:06.414471    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:13.414890 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:13.414857    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:13.427933 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:13.427907    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:14.078671 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:14.078637    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:54:17.727203 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:17.727169    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:54:18.855117 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:54:18.855084    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:55:36.350709 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.350673    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:55:36.358811 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.358779    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"]
Jul 29 10:55:36.368689 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:55:36.368654    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:55:36.368689 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:55:36.368689 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 10:55:36.368689 ip-10-0-129-80 kubenswrapper[2569]:  > containerID="209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086" cmd=["/bin/sleep","15"]
Jul 29 10:55:36.369021 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.368705    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp" message="" err=<
Jul 29 10:55:36.369021 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:55:36.369021 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 10:55:36.369021 ip-10-0-129-80 kubenswrapper[2569]:  >
Jul 29 10:55:36.369021 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:55:36.368722    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:55:36.369021 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:55:36.369021 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 10:55:36.369021 ip-10-0-129-80 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp" podUID="e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" containerName="main" containerID="cri-o://209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086"
Jul 29 10:55:36.369021 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.368767    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp" podUID="e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" containerName="main" containerID="cri-o://209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086" gracePeriod=60
Jul 29 10:55:36.725845 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.725821    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:55:36.850630 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.850597    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tls-certs\") pod \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") "
Jul 29 10:55:36.850819 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.850651    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67ffg\" (UniqueName: \"kubernetes.io/projected/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-kube-api-access-67ffg\") pod \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") "
Jul 29 10:55:36.850819 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.850685    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tmp-dir\") pod \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") "
Jul 29 10:55:36.850819 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.850712    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-home\") pod \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") "
Jul 29 10:55:36.850819 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.850731    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-model-cache\") pod \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") "
Jul 29 10:55:36.850819 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.850758    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-dshm\") pod \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\" (UID: \"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25\") "
Jul 29 10:55:36.851089 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.850999    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" (UID: "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:36.851089 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.851006    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-home" (OuterVolumeSpecName: "home") pod "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" (UID: "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:36.851089 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.851027    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" (UID: "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:36.853036 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.853010    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-dshm" (OuterVolumeSpecName: "dshm") pod "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" (UID: "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:36.853133 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.853031    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" (UID: "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:55:36.853133 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.853041    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-kube-api-access-67ffg" (OuterVolumeSpecName: "kube-api-access-67ffg") pod "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" (UID: "e2f6b58c-f9cc-4b09-8510-9ab40d91ef25"). InnerVolumeSpecName "kube-api-access-67ffg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:55:36.952488 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.952391    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:55:36.952488 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.952421    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67ffg\" (UniqueName: \"kubernetes.io/projected/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-kube-api-access-67ffg\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:55:36.952488 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.952454    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:55:36.952488 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.952463    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:55:36.952488 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.952471    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:55:36.952488 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:36.952479    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:55:37.468524 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.468488    2569 generic.go:356] "Generic (PLEG): container finished" podID="e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" containerID="209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086" exitCode=0
Jul 29 10:55:37.469013 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.468536    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"
Jul 29 10:55:37.469013 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.468584    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp" event={"ID":"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25","Type":"ContainerDied","Data":"209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086"}
Jul 29 10:55:37.469013 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.468648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp" event={"ID":"e2f6b58c-f9cc-4b09-8510-9ab40d91ef25","Type":"ContainerDied","Data":"fde9c60ccb7f3e8eb73278482cd5fbccc612a2245ec1501055fbdc80f9aabb7b"}
Jul 29 10:55:37.469013 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.468687    2569 scope.go:117] "RemoveContainer" containerID="209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086"
Jul 29 10:55:37.478169 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.478148    2569 scope.go:117] "RemoveContainer" containerID="209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086"
Jul 29 10:55:37.478458 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:55:37.478412    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086\": container with ID starting with 209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086 not found: ID does not exist" containerID="209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086"
Jul 29 10:55:37.478539 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.478460    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086"} err="failed to get container status \"209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086\": rpc error: code = NotFound desc = could not find container \"209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086\": container with ID starting with 209f8716abd13915245442de146aa92f1513507eb7d65fce3d216906bc91b086 not found: ID does not exist"
Jul 29 10:55:37.493186 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.493161    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"]
Jul 29 10:55:37.496039 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.496012    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-56cfp"]
Jul 29 10:55:37.838050 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:37.838018    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:55:38.043349 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:38.043313    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" path="/var/lib/kubelet/pods/e2f6b58c-f9cc-4b09-8510-9ab40d91ef25/volumes"
Jul 29 10:55:56.254138 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:56.254099    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:55:56.348604 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:55:56.348568    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:02.419652 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:02.419607    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:07.778093 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.778062    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:07.875186 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.875142    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:07.982074 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.982035    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"]
Jul 29 10:56:07.982739 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.982718    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" containerName="main"
Jul 29 10:56:07.982739 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.982739    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" containerName="main"
Jul 29 10:56:07.982906 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.982804    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f6b58c-f9cc-4b09-8510-9ab40d91ef25" containerName="main"
Jul 29 10:56:07.985676 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.985656    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:07.989229 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.989205    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 29 10:56:07.989943 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.989925    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 29 10:56:07.990051 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.989958    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 29 10:56:07.990518 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:07.990502    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-2kl8q\""
Jul 29 10:56:08.003123 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.003095    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"]
Jul 29 10:56:08.027158 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.027123    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:08.130344 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.130307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.130565 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.130395    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.130565 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.130491    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmptz\" (UniqueName: \"kubernetes.io/projected/183c981a-ec06-423f-91b9-c01464a3318b-kube-api-access-lmptz\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.130565 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.130558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183c981a-ec06-423f-91b9-c01464a3318b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.130697 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.130620    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.130697 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.130670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.231498 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.231457    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.231687 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.231519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmptz\" (UniqueName: \"kubernetes.io/projected/183c981a-ec06-423f-91b9-c01464a3318b-kube-api-access-lmptz\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.231687 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.231565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183c981a-ec06-423f-91b9-c01464a3318b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.231687 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.231607    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.231687 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.231656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.231915 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.231741    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.231972 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.231957    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.232029 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.231966    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.232081 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.232028    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.233969 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.233942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.234274 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.234256    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183c981a-ec06-423f-91b9-c01464a3318b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.240858 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.240829    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:08.240991 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.240971    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmptz\" (UniqueName: \"kubernetes.io/projected/183c981a-ec06-423f-91b9-c01464a3318b-kube-api-access-lmptz\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.296245 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.296209    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:08.435052 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.434925    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"]
Jul 29 10:56:08.438386 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:56:08.438354    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183c981a_ec06_423f_91b9_c01464a3318b.slice/crio-2751e60b3a9cdd319095376d78f57dea55dda1a4b459a52218f248d1615dee18 WatchSource:0}: Error finding container 2751e60b3a9cdd319095376d78f57dea55dda1a4b459a52218f248d1615dee18: Status 404 returned error can't find the container with id 2751e60b3a9cdd319095376d78f57dea55dda1a4b459a52218f248d1615dee18
Jul 29 10:56:08.620515 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.620472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c" event={"ID":"183c981a-ec06-423f-91b9-c01464a3318b","Type":"ContainerStarted","Data":"71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378"}
Jul 29 10:56:08.620723 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.620547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c" event={"ID":"183c981a-ec06-423f-91b9-c01464a3318b","Type":"ContainerStarted","Data":"2751e60b3a9cdd319095376d78f57dea55dda1a4b459a52218f248d1615dee18"}
Jul 29 10:56:08.637566 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:08.637466    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c" podStartSLOduration=1.6374170000000001 podStartE2EDuration="1.637417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:56:08.63700184 +0000 UTC m=+1643.030214711" watchObservedRunningTime="2026-07-29 10:56:08.637417 +0000 UTC m=+1643.030629870"
Jul 29 10:56:09.297217 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:09.297177    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:18.297160 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:18.297120    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:18.310218 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:18.310188    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:18.682715 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:18.682681    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:22.661951 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:22.661913    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:41.988975 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:41.988928    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:42.010647 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.010604    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"]
Jul 29 10:56:42.021183 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:56:42.021143    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:56:42.021183 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:56:42.021183 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 10:56:42.021183 ip-10-0-129-80 kubenswrapper[2569]:  > containerID="71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378" cmd=["/bin/sleep","15"]
Jul 29 10:56:42.021610 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.021577    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c" message="" err=<
Jul 29 10:56:42.021610 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:56:42.021610 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 10:56:42.021610 ip-10-0-129-80 kubenswrapper[2569]:  >
Jul 29 10:56:42.021926 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:56:42.021610    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:56:42.021926 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:56:42.021926 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 10:56:42.021926 ip-10-0-129-80 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c" podUID="183c981a-ec06-423f-91b9-c01464a3318b" containerName="main" containerID="cri-o://71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378"
Jul 29 10:56:42.021926 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.021667    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c" podUID="183c981a-ec06-423f-91b9-c01464a3318b" containerName="main" containerID="cri-o://71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378" gracePeriod=60
Jul 29 10:56:42.094137 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.094098    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:56:42.370783 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.370759    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:42.437985 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.437952    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-home\") pod \"183c981a-ec06-423f-91b9-c01464a3318b\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") "
Jul 29 10:56:42.438183 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438005    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-tmp-dir\") pod \"183c981a-ec06-423f-91b9-c01464a3318b\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") "
Jul 29 10:56:42.438183 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438046    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-model-cache\") pod \"183c981a-ec06-423f-91b9-c01464a3318b\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") "
Jul 29 10:56:42.438183 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438069    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183c981a-ec06-423f-91b9-c01464a3318b-tls-certs\") pod \"183c981a-ec06-423f-91b9-c01464a3318b\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") "
Jul 29 10:56:42.438183 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438146    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmptz\" (UniqueName: \"kubernetes.io/projected/183c981a-ec06-423f-91b9-c01464a3318b-kube-api-access-lmptz\") pod \"183c981a-ec06-423f-91b9-c01464a3318b\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") "
Jul 29 10:56:42.438400 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438210    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-dshm\") pod \"183c981a-ec06-423f-91b9-c01464a3318b\" (UID: \"183c981a-ec06-423f-91b9-c01464a3318b\") "
Jul 29 10:56:42.438400 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438277    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-home" (OuterVolumeSpecName: "home") pod "183c981a-ec06-423f-91b9-c01464a3318b" (UID: "183c981a-ec06-423f-91b9-c01464a3318b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:42.438400 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438350    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-model-cache" (OuterVolumeSpecName: "model-cache") pod "183c981a-ec06-423f-91b9-c01464a3318b" (UID: "183c981a-ec06-423f-91b9-c01464a3318b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:42.438400 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438363    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "183c981a-ec06-423f-91b9-c01464a3318b" (UID: "183c981a-ec06-423f-91b9-c01464a3318b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:42.438645 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438542    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.438645 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438561    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.438645 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.438576    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.440397 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.440375    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183c981a-ec06-423f-91b9-c01464a3318b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "183c981a-ec06-423f-91b9-c01464a3318b" (UID: "183c981a-ec06-423f-91b9-c01464a3318b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:56:42.440764 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.440740    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-dshm" (OuterVolumeSpecName: "dshm") pod "183c981a-ec06-423f-91b9-c01464a3318b" (UID: "183c981a-ec06-423f-91b9-c01464a3318b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:42.440857 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.440771    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183c981a-ec06-423f-91b9-c01464a3318b-kube-api-access-lmptz" (OuterVolumeSpecName: "kube-api-access-lmptz") pod "183c981a-ec06-423f-91b9-c01464a3318b" (UID: "183c981a-ec06-423f-91b9-c01464a3318b"). InnerVolumeSpecName "kube-api-access-lmptz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:56:42.540030 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.539945    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183c981a-ec06-423f-91b9-c01464a3318b-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.540030 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.539975    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmptz\" (UniqueName: \"kubernetes.io/projected/183c981a-ec06-423f-91b9-c01464a3318b-kube-api-access-lmptz\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.540030 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.539987    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183c981a-ec06-423f-91b9-c01464a3318b-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.790253 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.790163    2569 generic.go:356] "Generic (PLEG): container finished" podID="183c981a-ec06-423f-91b9-c01464a3318b" containerID="71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378" exitCode=0
Jul 29 10:56:42.790253 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.790213    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"
Jul 29 10:56:42.790456 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.790255    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c" event={"ID":"183c981a-ec06-423f-91b9-c01464a3318b","Type":"ContainerDied","Data":"71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378"}
Jul 29 10:56:42.790456 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.790309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c" event={"ID":"183c981a-ec06-423f-91b9-c01464a3318b","Type":"ContainerDied","Data":"2751e60b3a9cdd319095376d78f57dea55dda1a4b459a52218f248d1615dee18"}
Jul 29 10:56:42.790456 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.790332    2569 scope.go:117] "RemoveContainer" containerID="71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378"
Jul 29 10:56:42.800138 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.800120    2569 scope.go:117] "RemoveContainer" containerID="71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378"
Jul 29 10:56:42.800410 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:56:42.800372    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378\": container with ID starting with 71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378 not found: ID does not exist" containerID="71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378"
Jul 29 10:56:42.800528 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.800416    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378"} err="failed to get container status \"71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378\": rpc error: code = NotFound desc = could not find container \"71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378\": container with ID starting with 71a02f6177b8aa6fae5a4262d579ac14fb2c84dec6a6115f11f1631fc6fa0378 not found: ID does not exist"
Jul 29 10:56:42.828143 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.828117    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"]
Jul 29 10:56:42.831416 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:42.831385    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx6g6c"]
Jul 29 10:56:44.042145 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:56:44.042109    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="183c981a-ec06-423f-91b9-c01464a3318b" path="/var/lib/kubelet/pods/183c981a-ec06-423f-91b9-c01464a3318b/volumes"
Jul 29 10:57:01.642705 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:01.642666    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:01.741298 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:01.741249    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:07.617724 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:07.617656    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:08.090780 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.090744    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:08.367126 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.366993    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:08.410810 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.410779    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"]
Jul 29 10:57:08.411218 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.411201    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="183c981a-ec06-423f-91b9-c01464a3318b" containerName="main"
Jul 29 10:57:08.411302 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.411221    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="183c981a-ec06-423f-91b9-c01464a3318b" containerName="main"
Jul 29 10:57:08.411361 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.411307    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="183c981a-ec06-423f-91b9-c01464a3318b" containerName="main"
Jul 29 10:57:08.414305 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.414285    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:08.418284 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.418261    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-gj6v8\""
Jul 29 10:57:08.418411 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.418286    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 29 10:57:08.418411 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.418313    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-2mbv4\""
Jul 29 10:57:08.418411 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.418286    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 29 10:57:08.418411 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.418272    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 29 10:57:08.422555 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.422533    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"]
Jul 29 10:57:08.450860 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.450830    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:08.588315 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.588279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgglm\" (UniqueName: \"kubernetes.io/projected/6ba2ba92-383b-4dfb-9d02-88dd75345d05-kube-api-access-xgglm\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4\" (UID: \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:08.588510 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.588339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2ba92-383b-4dfb-9d02-88dd75345d05-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4\" (UID: \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:08.689179 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.689084    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2ba92-383b-4dfb-9d02-88dd75345d05-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4\" (UID: \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:08.689667 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.689191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgglm\" (UniqueName: \"kubernetes.io/projected/6ba2ba92-383b-4dfb-9d02-88dd75345d05-kube-api-access-xgglm\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4\" (UID: \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:08.691559 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.691538    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2ba92-383b-4dfb-9d02-88dd75345d05-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4\" (UID: \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:08.697403 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.697380    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgglm\" (UniqueName: \"kubernetes.io/projected/6ba2ba92-383b-4dfb-9d02-88dd75345d05-kube-api-access-xgglm\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4\" (UID: \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:08.725389 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.725357    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:08.864155 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.864127    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"]
Jul 29 10:57:08.866919 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:57:08.866892    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba2ba92_383b_4dfb_9d02_88dd75345d05.slice/crio-2d7b04f9d91a3bd1a9b975a9dff32cbee3ab9b97f3f83e6b63d458267e0887d3 WatchSource:0}: Error finding container 2d7b04f9d91a3bd1a9b975a9dff32cbee3ab9b97f3f83e6b63d458267e0887d3: Status 404 returned error can't find the container with id 2d7b04f9d91a3bd1a9b975a9dff32cbee3ab9b97f3f83e6b63d458267e0887d3
Jul 29 10:57:08.916552 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:08.916523    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4" event={"ID":"6ba2ba92-383b-4dfb-9d02-88dd75345d05","Type":"ContainerStarted","Data":"2d7b04f9d91a3bd1a9b975a9dff32cbee3ab9b97f3f83e6b63d458267e0887d3"}
Jul 29 10:57:09.923268 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:09.923224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4" event={"ID":"6ba2ba92-383b-4dfb-9d02-88dd75345d05","Type":"ContainerStarted","Data":"14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87"}
Jul 29 10:57:09.923697 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:09.923387    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:09.941912 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:09.941857    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4" podStartSLOduration=1.941839763 podStartE2EDuration="1.941839763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:57:09.94008502 +0000 UTC m=+1704.333297891" watchObservedRunningTime="2026-07-29 10:57:09.941839763 +0000 UTC m=+1704.335052634"
Jul 29 10:57:13.671769 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:13.671726    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:13.771702 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:13.771667    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:19.638362 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:19.638305    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:40.930167 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:40.930135    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:57:50.772481 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:50.772416    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:57:50.827813 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:57:50.827781    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:19.464764 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:19.464727    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:19.482308 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:19.482260    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"]
Jul 29 10:58:19.533047 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:19.533006    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:20.929756 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:20.929720    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4" podUID="6ba2ba92-383b-4dfb-9d02-88dd75345d05" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:58:30.930625 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:30.930585    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4" podUID="6ba2ba92-383b-4dfb-9d02-88dd75345d05" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:58:34.502028 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:34.501991    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4" podUID="6ba2ba92-383b-4dfb-9d02-88dd75345d05" containerName="main" containerID="cri-o://14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87" gracePeriod=45
Jul 29 10:58:34.758484 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:34.758395    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:58:34.851485 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:34.851420    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgglm\" (UniqueName: \"kubernetes.io/projected/6ba2ba92-383b-4dfb-9d02-88dd75345d05-kube-api-access-xgglm\") pod \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\" (UID: \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\") "
Jul 29 10:58:34.851685 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:34.851573    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2ba92-383b-4dfb-9d02-88dd75345d05-tls-certs\") pod \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\" (UID: \"6ba2ba92-383b-4dfb-9d02-88dd75345d05\") "
Jul 29 10:58:34.853656 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:34.853623    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba2ba92-383b-4dfb-9d02-88dd75345d05-kube-api-access-xgglm" (OuterVolumeSpecName: "kube-api-access-xgglm") pod "6ba2ba92-383b-4dfb-9d02-88dd75345d05" (UID: "6ba2ba92-383b-4dfb-9d02-88dd75345d05"). InnerVolumeSpecName "kube-api-access-xgglm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:58:34.853778 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:34.853660    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba2ba92-383b-4dfb-9d02-88dd75345d05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ba2ba92-383b-4dfb-9d02-88dd75345d05" (UID: "6ba2ba92-383b-4dfb-9d02-88dd75345d05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:58:34.952752 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:34.952716    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgglm\" (UniqueName: \"kubernetes.io/projected/6ba2ba92-383b-4dfb-9d02-88dd75345d05-kube-api-access-xgglm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:58:34.952752 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:34.952745    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2ba92-383b-4dfb-9d02-88dd75345d05-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:58:35.336228 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:35.336190    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ba2ba92-383b-4dfb-9d02-88dd75345d05" containerID="14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87" exitCode=0
Jul 29 10:58:35.336405 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:35.336255    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"
Jul 29 10:58:35.336405 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:35.336260    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4" event={"ID":"6ba2ba92-383b-4dfb-9d02-88dd75345d05","Type":"ContainerDied","Data":"14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87"}
Jul 29 10:58:35.336405 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:35.336304    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4" event={"ID":"6ba2ba92-383b-4dfb-9d02-88dd75345d05","Type":"ContainerDied","Data":"2d7b04f9d91a3bd1a9b975a9dff32cbee3ab9b97f3f83e6b63d458267e0887d3"}
Jul 29 10:58:35.336405 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:35.336325    2569 scope.go:117] "RemoveContainer" containerID="14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87"
Jul 29 10:58:35.345166 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:35.345147    2569 scope.go:117] "RemoveContainer" containerID="14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87"
Jul 29 10:58:35.345469 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:58:35.345452    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87\": container with ID starting with 14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87 not found: ID does not exist" containerID="14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87"
Jul 29 10:58:35.345523 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:35.345480    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87"} err="failed to get container status \"14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87\": rpc error: code = NotFound desc = could not find container \"14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87\": container with ID starting with 14cda799a5339b744a5c40be8562682b6cd5da28f876983daa55d811ef43ed87 not found: ID does not exist"
Jul 29 10:58:35.357495 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:35.357470    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"]
Jul 29 10:58:35.361267 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:35.361238    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65dgjf4"]
Jul 29 10:58:36.043059 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:36.043028    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba2ba92-383b-4dfb-9d02-88dd75345d05" path="/var/lib/kubelet/pods/6ba2ba92-383b-4dfb-9d02-88dd75345d05/volumes"
Jul 29 10:58:39.223611 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:39.223534    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:39.320178 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:39.320140    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:45.386281 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:45.386235    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:46.112036 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.112003    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:46.344646 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.344613    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"]
Jul 29 10:58:46.345028 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.345008    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba2ba92-383b-4dfb-9d02-88dd75345d05" containerName="main"
Jul 29 10:58:46.345028 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.345023    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba2ba92-383b-4dfb-9d02-88dd75345d05" containerName="main"
Jul 29 10:58:46.345212 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.345085    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba2ba92-383b-4dfb-9d02-88dd75345d05" containerName="main"
Jul 29 10:58:46.349156 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.349134    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:58:46.351961 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.351923    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:46.353416 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.353383    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 29 10:58:46.353560 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.353531    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-vhmrs\""
Jul 29 10:58:46.353656 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.353639    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-wftkd\""
Jul 29 10:58:46.353736 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.353679    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 29 10:58:46.353794 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.353750    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 29 10:58:46.359870 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.359850    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"]
Jul 29 10:58:46.447729 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.447649    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:46.458328 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.458301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ba8adc-7532-4904-8ff5-f29634f32b6d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v\" (UID: \"53ba8adc-7532-4904-8ff5-f29634f32b6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:58:46.458478 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.458338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t88ng\" (UniqueName: \"kubernetes.io/projected/53ba8adc-7532-4904-8ff5-f29634f32b6d-kube-api-access-t88ng\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v\" (UID: \"53ba8adc-7532-4904-8ff5-f29634f32b6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:58:46.559074 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.559034    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ba8adc-7532-4904-8ff5-f29634f32b6d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v\" (UID: \"53ba8adc-7532-4904-8ff5-f29634f32b6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:58:46.559074 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.559080    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88ng\" (UniqueName: \"kubernetes.io/projected/53ba8adc-7532-4904-8ff5-f29634f32b6d-kube-api-access-t88ng\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v\" (UID: \"53ba8adc-7532-4904-8ff5-f29634f32b6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:58:46.561681 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.561660    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ba8adc-7532-4904-8ff5-f29634f32b6d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v\" (UID: \"53ba8adc-7532-4904-8ff5-f29634f32b6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:58:46.567535 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.567501    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t88ng\" (UniqueName: \"kubernetes.io/projected/53ba8adc-7532-4904-8ff5-f29634f32b6d-kube-api-access-t88ng\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v\" (UID: \"53ba8adc-7532-4904-8ff5-f29634f32b6d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:58:46.662641 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.662603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:58:46.801714 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:46.801690    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"]
Jul 29 10:58:46.803705 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:58:46.803678    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ba8adc_7532_4904_8ff5_f29634f32b6d.slice/crio-bd17b958461315a3cfeb7126b12fb6de9492aceaa1cee645f571d6b75f285df8 WatchSource:0}: Error finding container bd17b958461315a3cfeb7126b12fb6de9492aceaa1cee645f571d6b75f285df8: Status 404 returned error can't find the container with id bd17b958461315a3cfeb7126b12fb6de9492aceaa1cee645f571d6b75f285df8
Jul 29 10:58:47.400044 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:47.400006    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v" event={"ID":"53ba8adc-7532-4904-8ff5-f29634f32b6d","Type":"ContainerStarted","Data":"e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb"}
Jul 29 10:58:47.400240 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:47.400063    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v" event={"ID":"53ba8adc-7532-4904-8ff5-f29634f32b6d","Type":"ContainerStarted","Data":"bd17b958461315a3cfeb7126b12fb6de9492aceaa1cee645f571d6b75f285df8"}
Jul 29 10:58:47.400240 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:47.400084    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:58:47.417885 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:47.417837    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v" podStartSLOduration=1.417820078 podStartE2EDuration="1.417820078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:58:47.416777409 +0000 UTC m=+1801.809990279" watchObservedRunningTime="2026-07-29 10:58:47.417820078 +0000 UTC m=+1801.811032948"
Jul 29 10:58:50.840802 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:50.840768    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:58:50.906345 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:58:50.906313    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:03.267585 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:03.267551    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:18.406764 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:18.406735    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:59:24.589852 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:24.589816    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:24.594331 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:24.594301    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"]
Jul 29 10:59:24.652467 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:24.652410    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:28.406946 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:28.406909    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v" podUID="53ba8adc-7532-4904-8ff5-f29634f32b6d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:59:38.406506 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:38.406453    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v" podUID="53ba8adc-7532-4904-8ff5-f29634f32b6d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:59:39.615607 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:39.615550    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v" podUID="53ba8adc-7532-4904-8ff5-f29634f32b6d" containerName="main" containerID="cri-o://e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb" gracePeriod=45
Jul 29 10:59:39.875372 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:39.875311    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:59:40.031816 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.031773    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t88ng\" (UniqueName: \"kubernetes.io/projected/53ba8adc-7532-4904-8ff5-f29634f32b6d-kube-api-access-t88ng\") pod \"53ba8adc-7532-4904-8ff5-f29634f32b6d\" (UID: \"53ba8adc-7532-4904-8ff5-f29634f32b6d\") "
Jul 29 10:59:40.031995 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.031845    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ba8adc-7532-4904-8ff5-f29634f32b6d-tls-certs\") pod \"53ba8adc-7532-4904-8ff5-f29634f32b6d\" (UID: \"53ba8adc-7532-4904-8ff5-f29634f32b6d\") "
Jul 29 10:59:40.034054 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.034026    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ba8adc-7532-4904-8ff5-f29634f32b6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53ba8adc-7532-4904-8ff5-f29634f32b6d" (UID: "53ba8adc-7532-4904-8ff5-f29634f32b6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:59:40.034168 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.034123    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ba8adc-7532-4904-8ff5-f29634f32b6d-kube-api-access-t88ng" (OuterVolumeSpecName: "kube-api-access-t88ng") pod "53ba8adc-7532-4904-8ff5-f29634f32b6d" (UID: "53ba8adc-7532-4904-8ff5-f29634f32b6d"). InnerVolumeSpecName "kube-api-access-t88ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:59:40.133010 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.132914    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t88ng\" (UniqueName: \"kubernetes.io/projected/53ba8adc-7532-4904-8ff5-f29634f32b6d-kube-api-access-t88ng\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:59:40.133010 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.132947    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ba8adc-7532-4904-8ff5-f29634f32b6d-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 10:59:40.664689 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.664655    2569 generic.go:356] "Generic (PLEG): container finished" podID="53ba8adc-7532-4904-8ff5-f29634f32b6d" containerID="e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb" exitCode=0
Jul 29 10:59:40.664689 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.664692    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v" event={"ID":"53ba8adc-7532-4904-8ff5-f29634f32b6d","Type":"ContainerDied","Data":"e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb"}
Jul 29 10:59:40.665236 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.664704    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"
Jul 29 10:59:40.665236 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.664734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v" event={"ID":"53ba8adc-7532-4904-8ff5-f29634f32b6d","Type":"ContainerDied","Data":"bd17b958461315a3cfeb7126b12fb6de9492aceaa1cee645f571d6b75f285df8"}
Jul 29 10:59:40.665236 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.664755    2569 scope.go:117] "RemoveContainer" containerID="e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb"
Jul 29 10:59:40.673477 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.673456    2569 scope.go:117] "RemoveContainer" containerID="e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb"
Jul 29 10:59:40.673734 ip-10-0-129-80 kubenswrapper[2569]: E0729 10:59:40.673716    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb\": container with ID starting with e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb not found: ID does not exist" containerID="e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb"
Jul 29 10:59:40.673803 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.673743    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb"} err="failed to get container status \"e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb\": rpc error: code = NotFound desc = could not find container \"e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb\": container with ID starting with e44c9bd139371572d28062cfda3e7a10ac6f1dbd05a5b801d717b26ac09a73fb not found: ID does not exist"
Jul 29 10:59:40.683086 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.683061    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"]
Jul 29 10:59:40.686104 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:40.686080    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92ms7v"]
Jul 29 10:59:42.042262 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:42.042231    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53ba8adc-7532-4904-8ff5-f29634f32b6d" path="/var/lib/kubelet/pods/53ba8adc-7532-4904-8ff5-f29634f32b6d/volumes"
Jul 29 10:59:44.678924 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:44.678892    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:44.771958 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:44.771924    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:48.855031 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:48.854993    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:49.046845 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.046814    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"]
Jul 29 10:59:49.047182 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.047169    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ba8adc-7532-4904-8ff5-f29634f32b6d" containerName="main"
Jul 29 10:59:49.047232 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.047185    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ba8adc-7532-4904-8ff5-f29634f32b6d" containerName="main"
Jul 29 10:59:49.047267 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.047257    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ba8adc-7532-4904-8ff5-f29634f32b6d" containerName="main"
Jul 29 10:59:49.050078 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.050048    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 10:59:49.054180 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.054156    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-626tr\""
Jul 29 10:59:49.054297 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.054215    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-kkzzv\""
Jul 29 10:59:49.054297 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.054275    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 29 10:59:49.054297 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.054223    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 29 10:59:49.054436 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.054282    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 29 10:59:49.058181 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.058161    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"]
Jul 29 10:59:49.082073 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.082034    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:49.206855 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.206760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01417845-5cf2-4dfa-8292-56b9bc22ed9f-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr\" (UID: \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 10:59:49.207173 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.207142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j42m\" (UniqueName: \"kubernetes.io/projected/01417845-5cf2-4dfa-8292-56b9bc22ed9f-kube-api-access-2j42m\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr\" (UID: \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 10:59:49.308302 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.308267    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01417845-5cf2-4dfa-8292-56b9bc22ed9f-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr\" (UID: \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 10:59:49.308549 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.308315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j42m\" (UniqueName: \"kubernetes.io/projected/01417845-5cf2-4dfa-8292-56b9bc22ed9f-kube-api-access-2j42m\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr\" (UID: \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 10:59:49.311041 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.311007    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01417845-5cf2-4dfa-8292-56b9bc22ed9f-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr\" (UID: \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 10:59:49.316184 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.316160    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j42m\" (UniqueName: \"kubernetes.io/projected/01417845-5cf2-4dfa-8292-56b9bc22ed9f-kube-api-access-2j42m\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr\" (UID: \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 10:59:49.362192 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.362155    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 10:59:49.499217 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.499191    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"]
Jul 29 10:59:49.501313 ip-10-0-129-80 kubenswrapper[2569]: W0729 10:59:49.501284    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01417845_5cf2_4dfa_8292_56b9bc22ed9f.slice/crio-e6cdf7ebd55174ea014e3a77393bd49a8d7c91a70b35872e899abde00fe50d94 WatchSource:0}: Error finding container e6cdf7ebd55174ea014e3a77393bd49a8d7c91a70b35872e899abde00fe50d94: Status 404 returned error can't find the container with id e6cdf7ebd55174ea014e3a77393bd49a8d7c91a70b35872e899abde00fe50d94
Jul 29 10:59:49.710111 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.710070    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr" event={"ID":"01417845-5cf2-4dfa-8292-56b9bc22ed9f","Type":"ContainerStarted","Data":"f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375"}
Jul 29 10:59:49.710310 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.710134    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr" event={"ID":"01417845-5cf2-4dfa-8292-56b9bc22ed9f","Type":"ContainerStarted","Data":"e6cdf7ebd55174ea014e3a77393bd49a8d7c91a70b35872e899abde00fe50d94"}
Jul 29 10:59:49.710310 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.710224    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 10:59:49.728028 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:49.727977    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr" podStartSLOduration=0.727961307 podStartE2EDuration="727.961307ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:59:49.727656619 +0000 UTC m=+1864.120869489" watchObservedRunningTime="2026-07-29 10:59:49.727961307 +0000 UTC m=+1864.121174176"
Jul 29 10:59:51.627158 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:51.627126    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:55.753649 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:55.753611    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:55.807961 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:55.807923    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:57.202643 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:57.202603    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 10:59:57.303694 ip-10-0-129-80 kubenswrapper[2569]: I0729 10:59:57.303649    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:00.117961 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:00.117923    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:01.788600 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:01.788569    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:01.890108 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:01.890069    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:20.716527 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:20.716451    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 11:00:22.069224 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:22.069183    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"]
Jul 29 11:00:22.115205 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:22.115171    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:22.212848 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:22.212807    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:30.716851 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:30.716811    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr" podUID="01417845-5cf2-4dfa-8292-56b9bc22ed9f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:00:37.090276 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.090228    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr" podUID="01417845-5cf2-4dfa-8292-56b9bc22ed9f" containerName="main" containerID="cri-o://f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375" gracePeriod=45
Jul 29 11:00:37.343808 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.343736    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 11:00:37.451608 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.451568    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01417845-5cf2-4dfa-8292-56b9bc22ed9f-tls-certs\") pod \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\" (UID: \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\") "
Jul 29 11:00:37.451608 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.451615    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j42m\" (UniqueName: \"kubernetes.io/projected/01417845-5cf2-4dfa-8292-56b9bc22ed9f-kube-api-access-2j42m\") pod \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\" (UID: \"01417845-5cf2-4dfa-8292-56b9bc22ed9f\") "
Jul 29 11:00:37.453859 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.453835    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01417845-5cf2-4dfa-8292-56b9bc22ed9f-kube-api-access-2j42m" (OuterVolumeSpecName: "kube-api-access-2j42m") pod "01417845-5cf2-4dfa-8292-56b9bc22ed9f" (UID: "01417845-5cf2-4dfa-8292-56b9bc22ed9f"). InnerVolumeSpecName "kube-api-access-2j42m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:00:37.453931 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.453895    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01417845-5cf2-4dfa-8292-56b9bc22ed9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01417845-5cf2-4dfa-8292-56b9bc22ed9f" (UID: "01417845-5cf2-4dfa-8292-56b9bc22ed9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:00:37.552307 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.552276    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01417845-5cf2-4dfa-8292-56b9bc22ed9f-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:00:37.552307 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.552302    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j42m\" (UniqueName: \"kubernetes.io/projected/01417845-5cf2-4dfa-8292-56b9bc22ed9f-kube-api-access-2j42m\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:00:37.945120 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.945086    2569 generic.go:356] "Generic (PLEG): container finished" podID="01417845-5cf2-4dfa-8292-56b9bc22ed9f" containerID="f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375" exitCode=0
Jul 29 11:00:37.945301 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.945130    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"
Jul 29 11:00:37.945301 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.945176    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr" event={"ID":"01417845-5cf2-4dfa-8292-56b9bc22ed9f","Type":"ContainerDied","Data":"f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375"}
Jul 29 11:00:37.945301 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.945229    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr" event={"ID":"01417845-5cf2-4dfa-8292-56b9bc22ed9f","Type":"ContainerDied","Data":"e6cdf7ebd55174ea014e3a77393bd49a8d7c91a70b35872e899abde00fe50d94"}
Jul 29 11:00:37.945301 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.945255    2569 scope.go:117] "RemoveContainer" containerID="f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375"
Jul 29 11:00:37.954813 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.954797    2569 scope.go:117] "RemoveContainer" containerID="f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375"
Jul 29 11:00:37.955049 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:00:37.955023    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375\": container with ID starting with f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375 not found: ID does not exist" containerID="f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375"
Jul 29 11:00:37.955115 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.955056    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375"} err="failed to get container status \"f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375\": rpc error: code = NotFound desc = could not find container \"f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375\": container with ID starting with f848e1b0415b348ecddab2336bd99f531797a5790f856c774f616077de5bc375 not found: ID does not exist"
Jul 29 11:00:37.967059 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.967034    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"]
Jul 29 11:00:37.970649 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:37.970617    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schesqlfr"]
Jul 29 11:00:38.042658 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:38.042624    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01417845-5cf2-4dfa-8292-56b9bc22ed9f" path="/var/lib/kubelet/pods/01417845-5cf2-4dfa-8292-56b9bc22ed9f/volumes"
Jul 29 11:00:42.123176 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:42.123143    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:46.050609 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:46.050566    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:48.237993 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:48.237955    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:53.775121 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:53.775083    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:53.877839 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:53.877809    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:58.054123 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.054088    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:58.450542 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.450509    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"]
Jul 29 11:00:58.450944 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.450927    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01417845-5cf2-4dfa-8292-56b9bc22ed9f" containerName="main"
Jul 29 11:00:58.450944 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.450945    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01417845-5cf2-4dfa-8292-56b9bc22ed9f" containerName="main"
Jul 29 11:00:58.451038 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.451006    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01417845-5cf2-4dfa-8292-56b9bc22ed9f" containerName="main"
Jul 29 11:00:58.454023 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.454005    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:00:58.458052 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.458031    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-9rr5h\""
Jul 29 11:00:58.458178 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.458059    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 29 11:00:58.458178 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.458034    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 29 11:00:58.458178 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.458034    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 29 11:00:58.458340 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.458324    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-qx479\""
Jul 29 11:00:58.462488 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.462466    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"]
Jul 29 11:00:58.473827 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.473798    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:00:58.542243 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.542197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdx52\" (UniqueName: \"kubernetes.io/projected/69a0f217-6741-40d2-ac36-e4240c343503-kube-api-access-bdx52\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v\" (UID: \"69a0f217-6741-40d2-ac36-e4240c343503\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:00:58.542469 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.542331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0f217-6741-40d2-ac36-e4240c343503-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v\" (UID: \"69a0f217-6741-40d2-ac36-e4240c343503\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:00:58.643835 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.643794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdx52\" (UniqueName: \"kubernetes.io/projected/69a0f217-6741-40d2-ac36-e4240c343503-kube-api-access-bdx52\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v\" (UID: \"69a0f217-6741-40d2-ac36-e4240c343503\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:00:58.644024 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.643898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0f217-6741-40d2-ac36-e4240c343503-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v\" (UID: \"69a0f217-6741-40d2-ac36-e4240c343503\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:00:58.646418 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.646391    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0f217-6741-40d2-ac36-e4240c343503-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v\" (UID: \"69a0f217-6741-40d2-ac36-e4240c343503\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:00:58.652273 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.652239    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdx52\" (UniqueName: \"kubernetes.io/projected/69a0f217-6741-40d2-ac36-e4240c343503-kube-api-access-bdx52\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v\" (UID: \"69a0f217-6741-40d2-ac36-e4240c343503\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:00:58.765071 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.764975    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:00:58.902341 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:58.902311    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"]
Jul 29 11:00:58.904491 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:00:58.904460    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a0f217_6741_40d2_ac36_e4240c343503.slice/crio-2a4ac7f85d56d9eab48501a2dd9cb5c424cf186b2f941860fe4357157b227b4e WatchSource:0}: Error finding container 2a4ac7f85d56d9eab48501a2dd9cb5c424cf186b2f941860fe4357157b227b4e: Status 404 returned error can't find the container with id 2a4ac7f85d56d9eab48501a2dd9cb5c424cf186b2f941860fe4357157b227b4e
Jul 29 11:00:59.053469 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:59.053352    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v" event={"ID":"69a0f217-6741-40d2-ac36-e4240c343503","Type":"ContainerStarted","Data":"61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3"}
Jul 29 11:00:59.053469 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:59.053406    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v" event={"ID":"69a0f217-6741-40d2-ac36-e4240c343503","Type":"ContainerStarted","Data":"2a4ac7f85d56d9eab48501a2dd9cb5c424cf186b2f941860fe4357157b227b4e"}
Jul 29 11:00:59.053469 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:59.053445    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:00:59.074758 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:00:59.074711    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v" podStartSLOduration=1.07469496 podStartE2EDuration="1.07469496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:00:59.073054002 +0000 UTC m=+1933.466266873" watchObservedRunningTime="2026-07-29 11:00:59.07469496 +0000 UTC m=+1933.467907830"
Jul 29 11:01:09.349706 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:09.349669    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:30.059953 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:30.059919    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:01:34.626951 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.626914    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"]
Jul 29 11:01:34.631106 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.631085    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.633924 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.633902    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 29 11:01:34.638300 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.638259    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"]
Jul 29 11:01:34.656995 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.656967    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:34.663129 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.663104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.663264 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.663137    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.663264 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.663219    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.663365 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.663283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.663365 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.663343    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b05fc-b7db-43e0-87f9-36dce29be3cb-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.663443 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.663378    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxm9\" (UniqueName: \"kubernetes.io/projected/525b05fc-b7db-43e0-87f9-36dce29be3cb-kube-api-access-8fxm9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.764584 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.764546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.764584 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.764584    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.764864 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.764628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.764864 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.764658    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.764864 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.764707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b05fc-b7db-43e0-87f9-36dce29be3cb-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.764864 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.764734    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxm9\" (UniqueName: \"kubernetes.io/projected/525b05fc-b7db-43e0-87f9-36dce29be3cb-kube-api-access-8fxm9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.765095 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.765042    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.765095 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.765082    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.765183 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.765151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.766902 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.766883    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.767184 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.767165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b05fc-b7db-43e0-87f9-36dce29be3cb-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.772902 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.772869    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxm9\" (UniqueName: \"kubernetes.io/projected/525b05fc-b7db-43e0-87f9-36dce29be3cb-kube-api-access-8fxm9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:34.888837 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.888751    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:34.943637 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:34.943605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:35.082780 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:35.082756    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"]
Jul 29 11:01:35.084648 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:01:35.084618    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525b05fc_b7db_43e0_87f9_36dce29be3cb.slice/crio-b1e484b6c5d4f4f4801d43bb155cebeeed3ff9bccb2bf816c961184de3277de9 WatchSource:0}: Error finding container b1e484b6c5d4f4f4801d43bb155cebeeed3ff9bccb2bf816c961184de3277de9: Status 404 returned error can't find the container with id b1e484b6c5d4f4f4801d43bb155cebeeed3ff9bccb2bf816c961184de3277de9
Jul 29 11:01:35.237207 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:35.237170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6" event={"ID":"525b05fc-b7db-43e0-87f9-36dce29be3cb","Type":"ContainerStarted","Data":"71ab5a836ad23f234722ef8abb58442ee56632a999fe468898fb86a12ba836ac"}
Jul 29 11:01:35.237416 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:35.237235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6" event={"ID":"525b05fc-b7db-43e0-87f9-36dce29be3cb","Type":"ContainerStarted","Data":"b1e484b6c5d4f4f4801d43bb155cebeeed3ff9bccb2bf816c961184de3277de9"}
Jul 29 11:01:35.254938 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:35.254882    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6" podStartSLOduration=1.254866011 podStartE2EDuration="1.254866011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:01:35.252855992 +0000 UTC m=+1969.646068875" watchObservedRunningTime="2026-07-29 11:01:35.254866011 +0000 UTC m=+1969.648078880"
Jul 29 11:01:35.943894 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:35.943857    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:41.705506 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:41.705392    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:01:44.944386 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:44.944350    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:44.957334 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:44.957305    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:01:45.296216 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:01:45.296129    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:02:08.124878 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.124836    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"]
Jul 29 11:02:08.135911 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:02:08.135867    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:02:08.135911 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:02:08.135911 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 11:02:08.135911 ip-10-0-129-80 kubenswrapper[2569]:  > containerID="71ab5a836ad23f234722ef8abb58442ee56632a999fe468898fb86a12ba836ac" cmd=["/bin/sleep","15"]
Jul 29 11:02:08.136242 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.136019    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6" message="" err=<
Jul 29 11:02:08.136242 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:02:08.136242 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 11:02:08.136242 ip-10-0-129-80 kubenswrapper[2569]:  >
Jul 29 11:02:08.136242 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:02:08.136056    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:02:08.136242 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:02:08.136242 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 11:02:08.136242 ip-10-0-129-80 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6" podUID="525b05fc-b7db-43e0-87f9-36dce29be3cb" containerName="main" containerID="cri-o://71ab5a836ad23f234722ef8abb58442ee56632a999fe468898fb86a12ba836ac"
Jul 29 11:02:08.136242 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.136109    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6" podUID="525b05fc-b7db-43e0-87f9-36dce29be3cb" containerName="main" containerID="cri-o://71ab5a836ad23f234722ef8abb58442ee56632a999fe468898fb86a12ba836ac" gracePeriod=60
Jul 29 11:02:08.136985 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.136869    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:08.207953 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.207917    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:08.285359 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.285324    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6" podUID="525b05fc-b7db-43e0-87f9-36dce29be3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 11:02:08.404420 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.404332    2569 generic.go:356] "Generic (PLEG): container finished" podID="525b05fc-b7db-43e0-87f9-36dce29be3cb" containerID="71ab5a836ad23f234722ef8abb58442ee56632a999fe468898fb86a12ba836ac" exitCode=0
Jul 29 11:02:08.404420 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.404403    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6" event={"ID":"525b05fc-b7db-43e0-87f9-36dce29be3cb","Type":"ContainerDied","Data":"71ab5a836ad23f234722ef8abb58442ee56632a999fe468898fb86a12ba836ac"}
Jul 29 11:02:08.893634 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.893608    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:02:08.985658 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.985620    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b05fc-b7db-43e0-87f9-36dce29be3cb-tls-certs\") pod \"525b05fc-b7db-43e0-87f9-36dce29be3cb\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") "
Jul 29 11:02:08.985826 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.985669    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-model-cache\") pod \"525b05fc-b7db-43e0-87f9-36dce29be3cb\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") "
Jul 29 11:02:08.985826 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.985699    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-tmp-dir\") pod \"525b05fc-b7db-43e0-87f9-36dce29be3cb\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") "
Jul 29 11:02:08.985826 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.985742    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fxm9\" (UniqueName: \"kubernetes.io/projected/525b05fc-b7db-43e0-87f9-36dce29be3cb-kube-api-access-8fxm9\") pod \"525b05fc-b7db-43e0-87f9-36dce29be3cb\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") "
Jul 29 11:02:08.985826 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.985823    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-home\") pod \"525b05fc-b7db-43e0-87f9-36dce29be3cb\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") "
Jul 29 11:02:08.986035 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.985878    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-dshm\") pod \"525b05fc-b7db-43e0-87f9-36dce29be3cb\" (UID: \"525b05fc-b7db-43e0-87f9-36dce29be3cb\") "
Jul 29 11:02:08.986035 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.985964    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "525b05fc-b7db-43e0-87f9-36dce29be3cb" (UID: "525b05fc-b7db-43e0-87f9-36dce29be3cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:02:08.986035 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.985981    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "525b05fc-b7db-43e0-87f9-36dce29be3cb" (UID: "525b05fc-b7db-43e0-87f9-36dce29be3cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:02:08.986189 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.986109    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-home" (OuterVolumeSpecName: "home") pod "525b05fc-b7db-43e0-87f9-36dce29be3cb" (UID: "525b05fc-b7db-43e0-87f9-36dce29be3cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:02:08.986248 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.986203    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:02:08.986248 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.986222    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:02:08.986248 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.986239    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:02:08.988068 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.988038    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-dshm" (OuterVolumeSpecName: "dshm") pod "525b05fc-b7db-43e0-87f9-36dce29be3cb" (UID: "525b05fc-b7db-43e0-87f9-36dce29be3cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:02:08.988204 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.988089    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525b05fc-b7db-43e0-87f9-36dce29be3cb-kube-api-access-8fxm9" (OuterVolumeSpecName: "kube-api-access-8fxm9") pod "525b05fc-b7db-43e0-87f9-36dce29be3cb" (UID: "525b05fc-b7db-43e0-87f9-36dce29be3cb"). InnerVolumeSpecName "kube-api-access-8fxm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:02:08.988204 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:08.988107    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525b05fc-b7db-43e0-87f9-36dce29be3cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "525b05fc-b7db-43e0-87f9-36dce29be3cb" (UID: "525b05fc-b7db-43e0-87f9-36dce29be3cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:02:09.087570 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:09.087513    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b05fc-b7db-43e0-87f9-36dce29be3cb-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:02:09.087756 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:09.087619    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b05fc-b7db-43e0-87f9-36dce29be3cb-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:02:09.087756 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:09.087640    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxm9\" (UniqueName: \"kubernetes.io/projected/525b05fc-b7db-43e0-87f9-36dce29be3cb-kube-api-access-8fxm9\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:02:09.412093 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:09.412004    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6" event={"ID":"525b05fc-b7db-43e0-87f9-36dce29be3cb","Type":"ContainerDied","Data":"b1e484b6c5d4f4f4801d43bb155cebeeed3ff9bccb2bf816c961184de3277de9"}
Jul 29 11:02:09.412093 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:09.412069    2569 scope.go:117] "RemoveContainer" containerID="71ab5a836ad23f234722ef8abb58442ee56632a999fe468898fb86a12ba836ac"
Jul 29 11:02:09.412682 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:09.412019    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"
Jul 29 11:02:09.433531 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:09.433504    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"]
Jul 29 11:02:09.439222 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:09.439196    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995v9gx6"]
Jul 29 11:02:10.042865 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:10.042823    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="525b05fc-b7db-43e0-87f9-36dce29be3cb" path="/var/lib/kubelet/pods/525b05fc-b7db-43e0-87f9-36dce29be3cb/volumes"
Jul 29 11:02:28.033043 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:28.033010    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"]
Jul 29 11:02:28.147159 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:28.147117    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:28.247659 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:28.247628    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:30.060342 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:30.060298    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v" podUID="69a0f217-6741-40d2-ac36-e4240c343503" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:02:40.060132 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:40.060085    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v" podUID="69a0f217-6741-40d2-ac36-e4240c343503" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:02:43.055624 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.055586    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v" podUID="69a0f217-6741-40d2-ac36-e4240c343503" containerName="main" containerID="cri-o://61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3" gracePeriod=45
Jul 29 11:02:43.309545 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.309481    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:02:43.395266 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.395230    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdx52\" (UniqueName: \"kubernetes.io/projected/69a0f217-6741-40d2-ac36-e4240c343503-kube-api-access-bdx52\") pod \"69a0f217-6741-40d2-ac36-e4240c343503\" (UID: \"69a0f217-6741-40d2-ac36-e4240c343503\") "
Jul 29 11:02:43.395481 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.395363    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0f217-6741-40d2-ac36-e4240c343503-tls-certs\") pod \"69a0f217-6741-40d2-ac36-e4240c343503\" (UID: \"69a0f217-6741-40d2-ac36-e4240c343503\") "
Jul 29 11:02:43.397267 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.397240    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a0f217-6741-40d2-ac36-e4240c343503-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69a0f217-6741-40d2-ac36-e4240c343503" (UID: "69a0f217-6741-40d2-ac36-e4240c343503"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:02:43.397381 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.397357    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a0f217-6741-40d2-ac36-e4240c343503-kube-api-access-bdx52" (OuterVolumeSpecName: "kube-api-access-bdx52") pod "69a0f217-6741-40d2-ac36-e4240c343503" (UID: "69a0f217-6741-40d2-ac36-e4240c343503"). InnerVolumeSpecName "kube-api-access-bdx52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:02:43.496734 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.496690    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a0f217-6741-40d2-ac36-e4240c343503-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:02:43.496734 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.496729    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdx52\" (UniqueName: \"kubernetes.io/projected/69a0f217-6741-40d2-ac36-e4240c343503-kube-api-access-bdx52\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:02:43.587142 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.587103    2569 generic.go:356] "Generic (PLEG): container finished" podID="69a0f217-6741-40d2-ac36-e4240c343503" containerID="61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3" exitCode=0
Jul 29 11:02:43.587302 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.587153    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"
Jul 29 11:02:43.587302 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.587185    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v" event={"ID":"69a0f217-6741-40d2-ac36-e4240c343503","Type":"ContainerDied","Data":"61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3"}
Jul 29 11:02:43.587302 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.587239    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v" event={"ID":"69a0f217-6741-40d2-ac36-e4240c343503","Type":"ContainerDied","Data":"2a4ac7f85d56d9eab48501a2dd9cb5c424cf186b2f941860fe4357157b227b4e"}
Jul 29 11:02:43.587302 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.587261    2569 scope.go:117] "RemoveContainer" containerID="61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3"
Jul 29 11:02:43.595960 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.595940    2569 scope.go:117] "RemoveContainer" containerID="61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3"
Jul 29 11:02:43.596203 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:02:43.596184    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3\": container with ID starting with 61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3 not found: ID does not exist" containerID="61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3"
Jul 29 11:02:43.596254 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.596211    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3"} err="failed to get container status \"61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3\": rpc error: code = NotFound desc = could not find container \"61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3\": container with ID starting with 61c81033045b0962d0b297789374d33f6bdf1ea5b216140b3f0bfe0a3f3b20b3 not found: ID does not exist"
Jul 29 11:02:43.608103 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.608079    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"]
Jul 29 11:02:43.613280 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:43.613260    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schertv8v"]
Jul 29 11:02:44.042168 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:44.042086    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a0f217-6741-40d2-ac36-e4240c343503" path="/var/lib/kubelet/pods/69a0f217-6741-40d2-ac36-e4240c343503/volumes"
Jul 29 11:02:48.020202 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:48.020168    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:48.118039 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:48.118006    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:53.066252 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.066222    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:53.251730 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.251671    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"]
Jul 29 11:02:53.252252 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.252225    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a0f217-6741-40d2-ac36-e4240c343503" containerName="main"
Jul 29 11:02:53.252252 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.252251    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a0f217-6741-40d2-ac36-e4240c343503" containerName="main"
Jul 29 11:02:53.252447 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.252290    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="525b05fc-b7db-43e0-87f9-36dce29be3cb" containerName="main"
Jul 29 11:02:53.252447 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.252299    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="525b05fc-b7db-43e0-87f9-36dce29be3cb" containerName="main"
Jul 29 11:02:53.252447 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.252386    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="525b05fc-b7db-43e0-87f9-36dce29be3cb" containerName="main"
Jul 29 11:02:53.252447 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.252402    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a0f217-6741-40d2-ac36-e4240c343503" containerName="main"
Jul 29 11:02:53.255461 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.255443    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:02:53.259455 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.259365    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 29 11:02:53.259455 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.259395    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-mtrm4\""
Jul 29 11:02:53.259455 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.259420    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-2rqx9\""
Jul 29 11:02:53.259689 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.259517    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 29 11:02:53.259689 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.259669    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 29 11:02:53.264510 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.264483    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"]
Jul 29 11:02:53.282980 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.282953    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:02:53.283127 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.283107    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610b9d7f-e459-481f-a05a-8031fe90962c-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh\" (UID: \"610b9d7f-e459-481f-a05a-8031fe90962c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:02:53.283176 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.283139    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szvp5\" (UniqueName: \"kubernetes.io/projected/610b9d7f-e459-481f-a05a-8031fe90962c-kube-api-access-szvp5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh\" (UID: \"610b9d7f-e459-481f-a05a-8031fe90962c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:02:53.383704 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.383662    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610b9d7f-e459-481f-a05a-8031fe90962c-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh\" (UID: \"610b9d7f-e459-481f-a05a-8031fe90962c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:02:53.383704 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.383708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szvp5\" (UniqueName: \"kubernetes.io/projected/610b9d7f-e459-481f-a05a-8031fe90962c-kube-api-access-szvp5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh\" (UID: \"610b9d7f-e459-481f-a05a-8031fe90962c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:02:53.386607 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.386578    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610b9d7f-e459-481f-a05a-8031fe90962c-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh\" (UID: \"610b9d7f-e459-481f-a05a-8031fe90962c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:02:53.391981 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.391954    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szvp5\" (UniqueName: \"kubernetes.io/projected/610b9d7f-e459-481f-a05a-8031fe90962c-kube-api-access-szvp5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh\" (UID: \"610b9d7f-e459-481f-a05a-8031fe90962c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:02:53.567885 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.567848    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:02:53.708801 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:53.708773    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"]
Jul 29 11:02:53.710982 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:02:53.710952    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610b9d7f_e459_481f_a05a_8031fe90962c.slice/crio-38fe05e466c20f21707bef2590a6bc58796e3a10aa92156ad9c2f226b7de277f WatchSource:0}: Error finding container 38fe05e466c20f21707bef2590a6bc58796e3a10aa92156ad9c2f226b7de277f: Status 404 returned error can't find the container with id 38fe05e466c20f21707bef2590a6bc58796e3a10aa92156ad9c2f226b7de277f
Jul 29 11:02:54.649771 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:54.649731    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh" event={"ID":"610b9d7f-e459-481f-a05a-8031fe90962c","Type":"ContainerStarted","Data":"b6fe49bf9090678455f1e3923216e55e54c1e614330b380eb310fcd57d1ce7ee"}
Jul 29 11:02:54.650179 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:54.649790    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh" event={"ID":"610b9d7f-e459-481f-a05a-8031fe90962c","Type":"ContainerStarted","Data":"38fe05e466c20f21707bef2590a6bc58796e3a10aa92156ad9c2f226b7de277f"}
Jul 29 11:02:54.650179 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:54.649839    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:02:54.666818 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:54.666762    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh" podStartSLOduration=1.666744541 podStartE2EDuration="1.666744541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:02:54.666707819 +0000 UTC m=+2049.059920690" watchObservedRunningTime="2026-07-29 11:02:54.666744541 +0000 UTC m=+2049.059957408"
Jul 29 11:02:54.751147 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:02:54.751111    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:00.342149 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:00.342109    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:00.399597 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:00.399558    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:03.409565 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:03.409519    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:13.231119 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:13.231040    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:13.331809 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:13.331771    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:25.656990 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:25.656957    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:03:35.917171 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:35.917117    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:39.723609 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.723572    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"]
Jul 29 11:03:39.727156 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.727135    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.729856 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.729828    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 29 11:03:39.737833 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.737807    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"]
Jul 29 11:03:39.759719 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.759690    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:39.896975 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.896936    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.897167 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.896991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.897167 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.897066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqvc5\" (UniqueName: \"kubernetes.io/projected/8755e19e-efbe-442f-9d36-c4aa58d0638c-kube-api-access-mqvc5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.897167 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.897114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.897301 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.897192    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8755e19e-efbe-442f-9d36-c4aa58d0638c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.897301 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.897259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.950374 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.950344    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:03:39.998523 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.998390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.998523 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.998460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.998523 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.998496    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.998818 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.998530    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqvc5\" (UniqueName: \"kubernetes.io/projected/8755e19e-efbe-442f-9d36-c4aa58d0638c-kube-api-access-mqvc5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.998818 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.998554    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.998818 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.998687    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8755e19e-efbe-442f-9d36-c4aa58d0638c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.998960 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.998870    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.998960 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.998914    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:39.998960 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:39.998947    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:40.000982 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:40.000951    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:40.001235 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:40.001218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8755e19e-efbe-442f-9d36-c4aa58d0638c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:40.006919 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:40.006893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqvc5\" (UniqueName: \"kubernetes.io/projected/8755e19e-efbe-442f-9d36-c4aa58d0638c-kube-api-access-mqvc5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:40.040996 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:40.040934    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:40.186387 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:40.186358    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"]
Jul 29 11:03:40.188979 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:03:40.188952    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8755e19e_efbe_442f_9d36_c4aa58d0638c.slice/crio-c7db6c6b8f330dc35babd0691b7a7e10fc678d76351f6b01f0d6623c46c2b69a WatchSource:0}: Error finding container c7db6c6b8f330dc35babd0691b7a7e10fc678d76351f6b01f0d6623c46c2b69a: Status 404 returned error can't find the container with id c7db6c6b8f330dc35babd0691b7a7e10fc678d76351f6b01f0d6623c46c2b69a
Jul 29 11:03:40.872716 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:40.872678    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b" event={"ID":"8755e19e-efbe-442f-9d36-c4aa58d0638c","Type":"ContainerStarted","Data":"8538aebd4d76e435388939f54a11b83137528ff4977aea95c67d474dcdce2e1e"}
Jul 29 11:03:40.873099 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:40.872741    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b" event={"ID":"8755e19e-efbe-442f-9d36-c4aa58d0638c","Type":"ContainerStarted","Data":"c7db6c6b8f330dc35babd0691b7a7e10fc678d76351f6b01f0d6623c46c2b69a"}
Jul 29 11:03:40.889547 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:40.889495    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b" podStartSLOduration=1.8894769660000001 podStartE2EDuration="1.889476966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:03:40.887753024 +0000 UTC m=+2095.280965895" watchObservedRunningTime="2026-07-29 11:03:40.889476966 +0000 UTC m=+2095.282689897"
Jul 29 11:03:41.041353 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:41.041316    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:50.043494 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:50.043459    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:50.054634 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:50.054609    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:50.934188 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:50.934159    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:03:52.515111 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:03:52.515077    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:04:12.840488 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:12.840457    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"]
Jul 29 11:04:12.850614 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:04:12.850576    2569 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:04:12.850614 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:04:12.850614 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 11:04:12.850614 ip-10-0-129-80 kubenswrapper[2569]:  > containerID="8538aebd4d76e435388939f54a11b83137528ff4977aea95c67d474dcdce2e1e" cmd=["/bin/sleep","15"]
Jul 29 11:04:12.850901 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:12.850633    2569 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b" message="" err=<
Jul 29 11:04:12.850901 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:04:12.850901 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 11:04:12.850901 ip-10-0-129-80 kubenswrapper[2569]:  >
Jul 29 11:04:12.850901 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:04:12.850651    2569 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:04:12.850901 ip-10-0-129-80 kubenswrapper[2569]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:04:12.850901 ip-10-0-129-80 kubenswrapper[2569]:         , stdout: , stderr: , exit code -1
Jul 29 11:04:12.850901 ip-10-0-129-80 kubenswrapper[2569]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b" podUID="8755e19e-efbe-442f-9d36-c4aa58d0638c" containerName="main" containerID="cri-o://8538aebd4d76e435388939f54a11b83137528ff4977aea95c67d474dcdce2e1e"
Jul 29 11:04:12.850901 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:12.850698    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b" podUID="8755e19e-efbe-442f-9d36-c4aa58d0638c" containerName="main" containerID="cri-o://8538aebd4d76e435388939f54a11b83137528ff4977aea95c67d474dcdce2e1e" gracePeriod=60
Jul 29 11:04:12.851377 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:12.851232    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:04:12.905182 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:12.905142    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:04:12.924033 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:12.923997    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b" podUID="8755e19e-efbe-442f-9d36-c4aa58d0638c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 29 11:04:13.030484 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.030438    2569 generic.go:356] "Generic (PLEG): container finished" podID="8755e19e-efbe-442f-9d36-c4aa58d0638c" containerID="8538aebd4d76e435388939f54a11b83137528ff4977aea95c67d474dcdce2e1e" exitCode=0
Jul 29 11:04:13.030484 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.030481    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b" event={"ID":"8755e19e-efbe-442f-9d36-c4aa58d0638c","Type":"ContainerDied","Data":"8538aebd4d76e435388939f54a11b83137528ff4977aea95c67d474dcdce2e1e"}
Jul 29 11:04:13.604590 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.604565    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:04:13.703255 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703216    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqvc5\" (UniqueName: \"kubernetes.io/projected/8755e19e-efbe-442f-9d36-c4aa58d0638c-kube-api-access-mqvc5\") pod \"8755e19e-efbe-442f-9d36-c4aa58d0638c\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") "
Jul 29 11:04:13.703255 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703262    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8755e19e-efbe-442f-9d36-c4aa58d0638c-tls-certs\") pod \"8755e19e-efbe-442f-9d36-c4aa58d0638c\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") "
Jul 29 11:04:13.703882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703292    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-dshm\") pod \"8755e19e-efbe-442f-9d36-c4aa58d0638c\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") "
Jul 29 11:04:13.703882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703320    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-model-cache\") pod \"8755e19e-efbe-442f-9d36-c4aa58d0638c\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") "
Jul 29 11:04:13.703882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703350    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-home\") pod \"8755e19e-efbe-442f-9d36-c4aa58d0638c\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") "
Jul 29 11:04:13.703882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703405    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-tmp-dir\") pod \"8755e19e-efbe-442f-9d36-c4aa58d0638c\" (UID: \"8755e19e-efbe-442f-9d36-c4aa58d0638c\") "
Jul 29 11:04:13.703882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703654    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-home" (OuterVolumeSpecName: "home") pod "8755e19e-efbe-442f-9d36-c4aa58d0638c" (UID: "8755e19e-efbe-442f-9d36-c4aa58d0638c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:04:13.703882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703666    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8755e19e-efbe-442f-9d36-c4aa58d0638c" (UID: "8755e19e-efbe-442f-9d36-c4aa58d0638c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:04:13.703882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703746    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8755e19e-efbe-442f-9d36-c4aa58d0638c" (UID: "8755e19e-efbe-442f-9d36-c4aa58d0638c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:04:13.703882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703806    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:04:13.703882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.703826    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:04:13.705698 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.705667    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-dshm" (OuterVolumeSpecName: "dshm") pod "8755e19e-efbe-442f-9d36-c4aa58d0638c" (UID: "8755e19e-efbe-442f-9d36-c4aa58d0638c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:04:13.705833 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.705706    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8755e19e-efbe-442f-9d36-c4aa58d0638c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8755e19e-efbe-442f-9d36-c4aa58d0638c" (UID: "8755e19e-efbe-442f-9d36-c4aa58d0638c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:04:13.705833 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.705773    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8755e19e-efbe-442f-9d36-c4aa58d0638c-kube-api-access-mqvc5" (OuterVolumeSpecName: "kube-api-access-mqvc5") pod "8755e19e-efbe-442f-9d36-c4aa58d0638c" (UID: "8755e19e-efbe-442f-9d36-c4aa58d0638c"). InnerVolumeSpecName "kube-api-access-mqvc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:04:13.804950 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.804861    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:04:13.804950 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.804892    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8755e19e-efbe-442f-9d36-c4aa58d0638c-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:04:13.804950 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.804902    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqvc5\" (UniqueName: \"kubernetes.io/projected/8755e19e-efbe-442f-9d36-c4aa58d0638c-kube-api-access-mqvc5\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:04:13.804950 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:13.804911    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8755e19e-efbe-442f-9d36-c4aa58d0638c-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:04:14.036338 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:14.036302    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"
Jul 29 11:04:14.042006 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:14.041976    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b" event={"ID":"8755e19e-efbe-442f-9d36-c4aa58d0638c","Type":"ContainerDied","Data":"c7db6c6b8f330dc35babd0691b7a7e10fc678d76351f6b01f0d6623c46c2b69a"}
Jul 29 11:04:14.042138 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:14.042035    2569 scope.go:117] "RemoveContainer" containerID="8538aebd4d76e435388939f54a11b83137528ff4977aea95c67d474dcdce2e1e"
Jul 29 11:04:14.060016 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:14.059954    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"]
Jul 29 11:04:14.061612 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:14.061591    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-tds5b"]
Jul 29 11:04:16.042032 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:16.041994    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8755e19e-efbe-442f-9d36-c4aa58d0638c" path="/var/lib/kubelet/pods/8755e19e-efbe-442f-9d36-c4aa58d0638c/volumes"
Jul 29 11:04:33.001574 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:33.001524    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"]
Jul 29 11:04:33.015412 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:33.015368    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:04:33.114383 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:33.114350    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:04:35.657021 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:35.656982    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh" podUID="610b9d7f-e459-481f-a05a-8031fe90962c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:04:45.656451 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:45.656399    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh" podUID="610b9d7f-e459-481f-a05a-8031fe90962c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:04:48.025652 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.025605    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh" podUID="610b9d7f-e459-481f-a05a-8031fe90962c" containerName="main" containerID="cri-o://b6fe49bf9090678455f1e3923216e55e54c1e614330b380eb310fcd57d1ce7ee" gracePeriod=45
Jul 29 11:04:48.196561 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.196472    2569 generic.go:356] "Generic (PLEG): container finished" podID="610b9d7f-e459-481f-a05a-8031fe90962c" containerID="b6fe49bf9090678455f1e3923216e55e54c1e614330b380eb310fcd57d1ce7ee" exitCode=0
Jul 29 11:04:48.196651 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.196572    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh" event={"ID":"610b9d7f-e459-481f-a05a-8031fe90962c","Type":"ContainerDied","Data":"b6fe49bf9090678455f1e3923216e55e54c1e614330b380eb310fcd57d1ce7ee"}
Jul 29 11:04:48.282250 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.282194    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:04:48.315025 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.314994    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szvp5\" (UniqueName: \"kubernetes.io/projected/610b9d7f-e459-481f-a05a-8031fe90962c-kube-api-access-szvp5\") pod \"610b9d7f-e459-481f-a05a-8031fe90962c\" (UID: \"610b9d7f-e459-481f-a05a-8031fe90962c\") "
Jul 29 11:04:48.315181 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.315111    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610b9d7f-e459-481f-a05a-8031fe90962c-tls-certs\") pod \"610b9d7f-e459-481f-a05a-8031fe90962c\" (UID: \"610b9d7f-e459-481f-a05a-8031fe90962c\") "
Jul 29 11:04:48.317285 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.317251    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610b9d7f-e459-481f-a05a-8031fe90962c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "610b9d7f-e459-481f-a05a-8031fe90962c" (UID: "610b9d7f-e459-481f-a05a-8031fe90962c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:04:48.317395 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.317333    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610b9d7f-e459-481f-a05a-8031fe90962c-kube-api-access-szvp5" (OuterVolumeSpecName: "kube-api-access-szvp5") pod "610b9d7f-e459-481f-a05a-8031fe90962c" (UID: "610b9d7f-e459-481f-a05a-8031fe90962c"). InnerVolumeSpecName "kube-api-access-szvp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:04:48.416740 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.416699    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szvp5\" (UniqueName: \"kubernetes.io/projected/610b9d7f-e459-481f-a05a-8031fe90962c-kube-api-access-szvp5\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:04:48.416740 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:48.416732    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610b9d7f-e459-481f-a05a-8031fe90962c-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:04:49.202385 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:49.202351    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"
Jul 29 11:04:49.202822 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:49.202355    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh" event={"ID":"610b9d7f-e459-481f-a05a-8031fe90962c","Type":"ContainerDied","Data":"38fe05e466c20f21707bef2590a6bc58796e3a10aa92156ad9c2f226b7de277f"}
Jul 29 11:04:49.202822 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:49.202499    2569 scope.go:117] "RemoveContainer" containerID="b6fe49bf9090678455f1e3923216e55e54c1e614330b380eb310fcd57d1ce7ee"
Jul 29 11:04:49.223395 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:49.223363    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"]
Jul 29 11:04:49.226199 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:49.226175    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche5tqnh"]
Jul 29 11:04:50.042338 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:50.042295    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610b9d7f-e459-481f-a05a-8031fe90962c" path="/var/lib/kubelet/pods/610b9d7f-e459-481f-a05a-8031fe90962c/volumes"
Jul 29 11:04:53.064039 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:53.063991    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:04:53.162207 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:53.162171    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:04:57.655813 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.655775    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:04:57.828714 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.828683    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"]
Jul 29 11:04:57.829085 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.829064    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610b9d7f-e459-481f-a05a-8031fe90962c" containerName="main"
Jul 29 11:04:57.829085 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.829079    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="610b9d7f-e459-481f-a05a-8031fe90962c" containerName="main"
Jul 29 11:04:57.829308 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.829243    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8755e19e-efbe-442f-9d36-c4aa58d0638c" containerName="main"
Jul 29 11:04:57.829308 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.829264    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8755e19e-efbe-442f-9d36-c4aa58d0638c" containerName="main"
Jul 29 11:04:57.829466 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.829409    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="610b9d7f-e459-481f-a05a-8031fe90962c" containerName="main"
Jul 29 11:04:57.829466 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.829451    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8755e19e-efbe-442f-9d36-c4aa58d0638c" containerName="main"
Jul 29 11:04:57.832497 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.832479    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:04:57.836269 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.836243    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 29 11:04:57.836439 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.836406    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 29 11:04:57.836514 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.836458    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-8lpxb\""
Jul 29 11:04:57.836514 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.836486    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-nl448\""
Jul 29 11:04:57.836633 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.836578    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 29 11:04:57.840585 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.840520    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"]
Jul 29 11:04:57.856509 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.856481    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:04:57.897795 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.897768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05620deb-df54-4380-8397-1d037eace294-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v\" (UID: \"05620deb-df54-4380-8397-1d037eace294\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:04:57.897936 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.897805    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4k6\" (UniqueName: \"kubernetes.io/projected/05620deb-df54-4380-8397-1d037eace294-kube-api-access-qg4k6\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v\" (UID: \"05620deb-df54-4380-8397-1d037eace294\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:04:57.998730 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.998649    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05620deb-df54-4380-8397-1d037eace294-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v\" (UID: \"05620deb-df54-4380-8397-1d037eace294\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:04:57.998730 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:57.998685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4k6\" (UniqueName: \"kubernetes.io/projected/05620deb-df54-4380-8397-1d037eace294-kube-api-access-qg4k6\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v\" (UID: \"05620deb-df54-4380-8397-1d037eace294\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:04:58.002017 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:58.001983    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05620deb-df54-4380-8397-1d037eace294-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v\" (UID: \"05620deb-df54-4380-8397-1d037eace294\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:04:58.007579 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:58.007552    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4k6\" (UniqueName: \"kubernetes.io/projected/05620deb-df54-4380-8397-1d037eace294-kube-api-access-qg4k6\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v\" (UID: \"05620deb-df54-4380-8397-1d037eace294\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:04:58.143669 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:58.143627    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:04:58.280716 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:58.279046    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"]
Jul 29 11:04:58.283620 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:04:58.283576    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05620deb_df54_4380_8397_1d037eace294.slice/crio-387dffe6b58add4693c4749e195fe40568d202e75c9c2c04bbb79c832e6365e0 WatchSource:0}: Error finding container 387dffe6b58add4693c4749e195fe40568d202e75c9c2c04bbb79c832e6365e0: Status 404 returned error can't find the container with id 387dffe6b58add4693c4749e195fe40568d202e75c9c2c04bbb79c832e6365e0
Jul 29 11:04:59.253743 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:59.253701    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v" event={"ID":"05620deb-df54-4380-8397-1d037eace294","Type":"ContainerStarted","Data":"5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2"}
Jul 29 11:04:59.254132 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:59.253768    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v" event={"ID":"05620deb-df54-4380-8397-1d037eace294","Type":"ContainerStarted","Data":"387dffe6b58add4693c4749e195fe40568d202e75c9c2c04bbb79c832e6365e0"}
Jul 29 11:04:59.254132 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:59.253795    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:04:59.271765 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:04:59.271710    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v" podStartSLOduration=2.271695254 podStartE2EDuration="2.271695254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:04:59.269205614 +0000 UTC m=+2173.662418480" watchObservedRunningTime="2026-07-29 11:04:59.271695254 +0000 UTC m=+2173.664908130"
Jul 29 11:05:00.042792 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:05:00.042758    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:04.674571 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:05:04.674531    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:04.780976 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:05:04.780939    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:17.221582 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:05:17.221545    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:05:30.260773 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:05:30.260741    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:05:46.876054 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:05:46.876011    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:06:42.169323 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:42.169239    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"]
Jul 29 11:06:42.195241 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:42.195205    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:06:42.294212 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:42.294165    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:06:50.260540 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:50.260501    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v" podUID="05620deb-df54-4380-8397-1d037eace294" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:06:57.190406 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.190334    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v" podUID="05620deb-df54-4380-8397-1d037eace294" containerName="main" containerID="cri-o://5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2" gracePeriod=45
Jul 29 11:06:57.448362 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.448301    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:06:57.555137 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.555103    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05620deb-df54-4380-8397-1d037eace294-tls-certs\") pod \"05620deb-df54-4380-8397-1d037eace294\" (UID: \"05620deb-df54-4380-8397-1d037eace294\") "
Jul 29 11:06:57.555313 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.555158    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg4k6\" (UniqueName: \"kubernetes.io/projected/05620deb-df54-4380-8397-1d037eace294-kube-api-access-qg4k6\") pod \"05620deb-df54-4380-8397-1d037eace294\" (UID: \"05620deb-df54-4380-8397-1d037eace294\") "
Jul 29 11:06:57.557290 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.557260    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05620deb-df54-4380-8397-1d037eace294-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05620deb-df54-4380-8397-1d037eace294" (UID: "05620deb-df54-4380-8397-1d037eace294"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:06:57.557404 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.557354    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05620deb-df54-4380-8397-1d037eace294-kube-api-access-qg4k6" (OuterVolumeSpecName: "kube-api-access-qg4k6") pod "05620deb-df54-4380-8397-1d037eace294" (UID: "05620deb-df54-4380-8397-1d037eace294"). InnerVolumeSpecName "kube-api-access-qg4k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:06:57.656829 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.656792    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05620deb-df54-4380-8397-1d037eace294-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:06:57.656829 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.656824    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4k6\" (UniqueName: \"kubernetes.io/projected/05620deb-df54-4380-8397-1d037eace294-kube-api-access-qg4k6\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:06:57.840529 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.840488    2569 generic.go:356] "Generic (PLEG): container finished" podID="05620deb-df54-4380-8397-1d037eace294" containerID="5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2" exitCode=0
Jul 29 11:06:57.840708 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.840539    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"
Jul 29 11:06:57.840708 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.840571    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v" event={"ID":"05620deb-df54-4380-8397-1d037eace294","Type":"ContainerDied","Data":"5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2"}
Jul 29 11:06:57.840708 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.840627    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v" event={"ID":"05620deb-df54-4380-8397-1d037eace294","Type":"ContainerDied","Data":"387dffe6b58add4693c4749e195fe40568d202e75c9c2c04bbb79c832e6365e0"}
Jul 29 11:06:57.840708 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.840651    2569 scope.go:117] "RemoveContainer" containerID="5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2"
Jul 29 11:06:57.850150 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.850125    2569 scope.go:117] "RemoveContainer" containerID="5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2"
Jul 29 11:06:57.850407 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:06:57.850387    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2\": container with ID starting with 5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2 not found: ID does not exist" containerID="5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2"
Jul 29 11:06:57.850511 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.850415    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2"} err="failed to get container status \"5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2\": rpc error: code = NotFound desc = could not find container \"5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2\": container with ID starting with 5effacea8986598ea61df512991fd91c6c1005a0e402c61a8e8d656056caf0c2 not found: ID does not exist"
Jul 29 11:06:57.862885 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.862861    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"]
Jul 29 11:06:57.866537 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:57.866517    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schecgb4v"]
Jul 29 11:06:58.041967 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:06:58.041936    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05620deb-df54-4380-8397-1d037eace294" path="/var/lib/kubelet/pods/05620deb-df54-4380-8397-1d037eace294/volumes"
Jul 29 11:07:02.116792 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:02.116759    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:07:02.226738 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:02.226693    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:07:07.972129 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:07.972091    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:07:12.253469 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.253413    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:07:12.486093 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.486057    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"]
Jul 29 11:07:12.486578 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.486555    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05620deb-df54-4380-8397-1d037eace294" containerName="main"
Jul 29 11:07:12.486578 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.486580    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05620deb-df54-4380-8397-1d037eace294" containerName="main"
Jul 29 11:07:12.486722 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.486671    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05620deb-df54-4380-8397-1d037eace294" containerName="main"
Jul 29 11:07:12.491156 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.491136    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:12.495150 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.495122    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-4sfzz\""
Jul 29 11:07:12.495279 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.495135    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 29 11:07:12.495279 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.495171    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-4b5bz\""
Jul 29 11:07:12.495525 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.495503    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 29 11:07:12.495645 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.495539    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 29 11:07:12.498008 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.497985    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"]
Jul 29 11:07:12.515021 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.514924    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:07:12.584747 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.584713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2qg\" (UniqueName: \"kubernetes.io/projected/51f12869-452c-4134-b8a0-f35da12fa53a-kube-api-access-nq2qg\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9\" (UID: \"51f12869-452c-4134-b8a0-f35da12fa53a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:12.584920 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.584776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f12869-452c-4134-b8a0-f35da12fa53a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9\" (UID: \"51f12869-452c-4134-b8a0-f35da12fa53a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:12.686062 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.686017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2qg\" (UniqueName: \"kubernetes.io/projected/51f12869-452c-4134-b8a0-f35da12fa53a-kube-api-access-nq2qg\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9\" (UID: \"51f12869-452c-4134-b8a0-f35da12fa53a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:12.686253 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.686082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f12869-452c-4134-b8a0-f35da12fa53a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9\" (UID: \"51f12869-452c-4134-b8a0-f35da12fa53a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:12.688696 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.688664    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f12869-452c-4134-b8a0-f35da12fa53a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9\" (UID: \"51f12869-452c-4134-b8a0-f35da12fa53a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:12.693812 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.693789    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2qg\" (UniqueName: \"kubernetes.io/projected/51f12869-452c-4134-b8a0-f35da12fa53a-kube-api-access-nq2qg\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9\" (UID: \"51f12869-452c-4134-b8a0-f35da12fa53a\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:12.804047 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.803952    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:12.952444 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:12.952401    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"]
Jul 29 11:07:12.954679 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:07:12.954650    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f12869_452c_4134_b8a0_f35da12fa53a.slice/crio-34bea3e49d3fc57287b2f55430ad9b89fbcc134ebdbede65d32d4a21354ec5e0 WatchSource:0}: Error finding container 34bea3e49d3fc57287b2f55430ad9b89fbcc134ebdbede65d32d4a21354ec5e0: Status 404 returned error can't find the container with id 34bea3e49d3fc57287b2f55430ad9b89fbcc134ebdbede65d32d4a21354ec5e0
Jul 29 11:07:13.918853 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:13.918812    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9" event={"ID":"51f12869-452c-4134-b8a0-f35da12fa53a","Type":"ContainerStarted","Data":"5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177"}
Jul 29 11:07:13.919337 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:13.918867    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9" event={"ID":"51f12869-452c-4134-b8a0-f35da12fa53a","Type":"ContainerStarted","Data":"34bea3e49d3fc57287b2f55430ad9b89fbcc134ebdbede65d32d4a21354ec5e0"}
Jul 29 11:07:13.919337 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:13.918887    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:13.936418 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:13.936363    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9" podStartSLOduration=1.9363470390000002 podStartE2EDuration="1.936347039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:07:13.934820675 +0000 UTC m=+2308.328033559" watchObservedRunningTime="2026-07-29 11:07:13.936347039 +0000 UTC m=+2308.329559909"
Jul 29 11:07:14.271836 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:14.271740    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:07:14.372523 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:14.372492    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:07:27.757260 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:27.757224    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:07:44.926316 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:44.926286    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:07:49.877805 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:07:49.877772    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:28.187170 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:28.187129    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"]
Jul 29 11:08:28.191542 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:28.191509    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:28.246790 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:28.246738    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:33.810021 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:33.809984    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"]
Jul 29 11:08:33.813069 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:33.813037    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:33.877122 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:33.877086    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n" podUID="971bc44f-98be-4b97-801f-cbabc9c334fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:08:33.912914 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:33.912880    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:34.925977 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:34.925939    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9" podUID="51f12869-452c-4134-b8a0-f35da12fa53a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:08:43.207595 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.207528    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n" podUID="971bc44f-98be-4b97-801f-cbabc9c334fe" containerName="main" containerID="cri-o://1c28763794c1ab583920e367774678ee3607ec79d8dbd92be025450c97f8cd46" gracePeriod=45
Jul 29 11:08:43.358167 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.358133    2569 generic.go:356] "Generic (PLEG): container finished" podID="971bc44f-98be-4b97-801f-cbabc9c334fe" containerID="1c28763794c1ab583920e367774678ee3607ec79d8dbd92be025450c97f8cd46" exitCode=0
Jul 29 11:08:43.358167 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.358169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n" event={"ID":"971bc44f-98be-4b97-801f-cbabc9c334fe","Type":"ContainerDied","Data":"1c28763794c1ab583920e367774678ee3607ec79d8dbd92be025450c97f8cd46"}
Jul 29 11:08:43.468141 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.468078    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 11:08:43.580810 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.580773    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971bc44f-98be-4b97-801f-cbabc9c334fe-tls-certs\") pod \"971bc44f-98be-4b97-801f-cbabc9c334fe\" (UID: \"971bc44f-98be-4b97-801f-cbabc9c334fe\") "
Jul 29 11:08:43.580984 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.580875    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/971bc44f-98be-4b97-801f-cbabc9c334fe-kube-api-access-vmgwz\") pod \"971bc44f-98be-4b97-801f-cbabc9c334fe\" (UID: \"971bc44f-98be-4b97-801f-cbabc9c334fe\") "
Jul 29 11:08:43.583042 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.583010    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971bc44f-98be-4b97-801f-cbabc9c334fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "971bc44f-98be-4b97-801f-cbabc9c334fe" (UID: "971bc44f-98be-4b97-801f-cbabc9c334fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:08:43.583042 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.583031    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971bc44f-98be-4b97-801f-cbabc9c334fe-kube-api-access-vmgwz" (OuterVolumeSpecName: "kube-api-access-vmgwz") pod "971bc44f-98be-4b97-801f-cbabc9c334fe" (UID: "971bc44f-98be-4b97-801f-cbabc9c334fe"). InnerVolumeSpecName "kube-api-access-vmgwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:08:43.682047 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.682014    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971bc44f-98be-4b97-801f-cbabc9c334fe-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:08:43.682047 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:43.682044    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmgwz\" (UniqueName: \"kubernetes.io/projected/971bc44f-98be-4b97-801f-cbabc9c334fe-kube-api-access-vmgwz\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:08:44.364301 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:44.364268    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"
Jul 29 11:08:44.364878 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:44.364307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n" event={"ID":"971bc44f-98be-4b97-801f-cbabc9c334fe","Type":"ContainerDied","Data":"1be0f958ddf557991f0f51493cf329bdd8e18f0854043c540275fdc758de2b5d"}
Jul 29 11:08:44.364878 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:44.364383    2569 scope.go:117] "RemoveContainer" containerID="1c28763794c1ab583920e367774678ee3607ec79d8dbd92be025450c97f8cd46"
Jul 29 11:08:44.380826 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:44.380796    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"]
Jul 29 11:08:44.384895 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:44.384869    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-nmc7n"]
Jul 29 11:08:44.926317 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:44.926279    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9" podUID="51f12869-452c-4134-b8a0-f35da12fa53a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:08:46.041551 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:46.041521    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971bc44f-98be-4b97-801f-cbabc9c334fe" path="/var/lib/kubelet/pods/971bc44f-98be-4b97-801f-cbabc9c334fe/volumes"
Jul 29 11:08:48.421188 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:48.421150    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:48.571966 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:48.571929    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:48.830776 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:48.830736    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9" podUID="51f12869-452c-4134-b8a0-f35da12fa53a" containerName="main" containerID="cri-o://5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177" gracePeriod=45
Jul 29 11:08:49.084543 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.084473    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:08:49.235099 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.235062    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f12869-452c-4134-b8a0-f35da12fa53a-tls-certs\") pod \"51f12869-452c-4134-b8a0-f35da12fa53a\" (UID: \"51f12869-452c-4134-b8a0-f35da12fa53a\") "
Jul 29 11:08:49.235099 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.235113    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq2qg\" (UniqueName: \"kubernetes.io/projected/51f12869-452c-4134-b8a0-f35da12fa53a-kube-api-access-nq2qg\") pod \"51f12869-452c-4134-b8a0-f35da12fa53a\" (UID: \"51f12869-452c-4134-b8a0-f35da12fa53a\") "
Jul 29 11:08:49.237718 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.237682    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f12869-452c-4134-b8a0-f35da12fa53a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51f12869-452c-4134-b8a0-f35da12fa53a" (UID: "51f12869-452c-4134-b8a0-f35da12fa53a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:08:49.237851 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.237746    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f12869-452c-4134-b8a0-f35da12fa53a-kube-api-access-nq2qg" (OuterVolumeSpecName: "kube-api-access-nq2qg") pod "51f12869-452c-4134-b8a0-f35da12fa53a" (UID: "51f12869-452c-4134-b8a0-f35da12fa53a"). InnerVolumeSpecName "kube-api-access-nq2qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:08:49.336177 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.336078    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq2qg\" (UniqueName: \"kubernetes.io/projected/51f12869-452c-4134-b8a0-f35da12fa53a-kube-api-access-nq2qg\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:08:49.336177 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.336113    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f12869-452c-4134-b8a0-f35da12fa53a-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:08:49.396576 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.396536    2569 generic.go:356] "Generic (PLEG): container finished" podID="51f12869-452c-4134-b8a0-f35da12fa53a" containerID="5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177" exitCode=0
Jul 29 11:08:49.396576 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.396580    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"
Jul 29 11:08:49.396791 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.396599    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9" event={"ID":"51f12869-452c-4134-b8a0-f35da12fa53a","Type":"ContainerDied","Data":"5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177"}
Jul 29 11:08:49.396791 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.396649    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9" event={"ID":"51f12869-452c-4134-b8a0-f35da12fa53a","Type":"ContainerDied","Data":"34bea3e49d3fc57287b2f55430ad9b89fbcc134ebdbede65d32d4a21354ec5e0"}
Jul 29 11:08:49.396791 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.396672    2569 scope.go:117] "RemoveContainer" containerID="5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177"
Jul 29 11:08:49.405540 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.405517    2569 scope.go:117] "RemoveContainer" containerID="5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177"
Jul 29 11:08:49.405867 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:08:49.405842    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177\": container with ID starting with 5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177 not found: ID does not exist" containerID="5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177"
Jul 29 11:08:49.405944 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.405874    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177"} err="failed to get container status \"5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177\": rpc error: code = NotFound desc = could not find container \"5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177\": container with ID starting with 5705e9aa06bd06277a8f51c453d3c99a990d886a70cc20887aa5489f352bc177 not found: ID does not exist"
Jul 29 11:08:49.418107 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.418076    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"]
Jul 29 11:08:49.424340 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:49.424314    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche76fk9"]
Jul 29 11:08:50.041673 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:50.041639    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f12869-452c-4134-b8a0-f35da12fa53a" path="/var/lib/kubelet/pods/51f12869-452c-4134-b8a0-f35da12fa53a/volumes"
Jul 29 11:08:53.897489 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:53.897455    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:53.994021 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:53.993982    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:54.088790 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:54.088749    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:54.275822 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:54.275737    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:56.501871 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:56.501834    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:56.589592 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:56.589559    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:57.391327 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:57.391295    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:59.176884 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.176836    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:59.519645 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.519552    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"]
Jul 29 11:08:59.520216 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.520189    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f12869-452c-4134-b8a0-f35da12fa53a" containerName="main"
Jul 29 11:08:59.520534 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.520507    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f12869-452c-4134-b8a0-f35da12fa53a" containerName="main"
Jul 29 11:08:59.520668 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.520581    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971bc44f-98be-4b97-801f-cbabc9c334fe" containerName="main"
Jul 29 11:08:59.520668 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.520592    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="971bc44f-98be-4b97-801f-cbabc9c334fe" containerName="main"
Jul 29 11:08:59.520793 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.520693    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f12869-452c-4134-b8a0-f35da12fa53a" containerName="main"
Jul 29 11:08:59.520793 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.520704    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="971bc44f-98be-4b97-801f-cbabc9c334fe" containerName="main"
Jul 29 11:08:59.523795 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.523775    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:08:59.527713 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.527678    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-rsqwv\""
Jul 29 11:08:59.527713 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.527681    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 29 11:08:59.527895 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.527747    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-epp-sa-dockercfg-h6wm9\""
Jul 29 11:08:59.527895 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.527849    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 29 11:08:59.527997 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.527938    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 29 11:08:59.531789 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.531733    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"]
Jul 29 11:08:59.620363 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.620318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d50e7-17a7-4a76-97a2-a046205a13f3-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r\" (UID: \"b96d50e7-17a7-4a76-97a2-a046205a13f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:08:59.620582 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.620416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/b96d50e7-17a7-4a76-97a2-a046205a13f3-kube-api-access-f6hrl\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r\" (UID: \"b96d50e7-17a7-4a76-97a2-a046205a13f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:08:59.627092 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.627062    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:59.721334 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.721291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/b96d50e7-17a7-4a76-97a2-a046205a13f3-kube-api-access-f6hrl\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r\" (UID: \"b96d50e7-17a7-4a76-97a2-a046205a13f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:08:59.721547 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.721359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d50e7-17a7-4a76-97a2-a046205a13f3-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r\" (UID: \"b96d50e7-17a7-4a76-97a2-a046205a13f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:08:59.723946 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.723921    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d50e7-17a7-4a76-97a2-a046205a13f3-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r\" (UID: \"b96d50e7-17a7-4a76-97a2-a046205a13f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:08:59.727622 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.727593    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:08:59.729281 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.729260    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/b96d50e7-17a7-4a76-97a2-a046205a13f3-kube-api-access-f6hrl\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r\" (UID: \"b96d50e7-17a7-4a76-97a2-a046205a13f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:08:59.835495 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.835469    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:08:59.985028 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:08:59.985004    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"]
Jul 29 11:08:59.987366 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:08:59.987334    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96d50e7_17a7_4a76_97a2_a046205a13f3.slice/crio-9a11edc39a39f822885015dffb8dbe1317ae30c1b555a1627922bc4fb4b70463 WatchSource:0}: Error finding container 9a11edc39a39f822885015dffb8dbe1317ae30c1b555a1627922bc4fb4b70463: Status 404 returned error can't find the container with id 9a11edc39a39f822885015dffb8dbe1317ae30c1b555a1627922bc4fb4b70463
Jul 29 11:09:00.448827 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:00.448796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r" event={"ID":"b96d50e7-17a7-4a76-97a2-a046205a13f3","Type":"ContainerStarted","Data":"757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db"}
Jul 29 11:09:00.449216 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:00.448847    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r" event={"ID":"b96d50e7-17a7-4a76-97a2-a046205a13f3","Type":"ContainerStarted","Data":"9a11edc39a39f822885015dffb8dbe1317ae30c1b555a1627922bc4fb4b70463"}
Jul 29 11:09:00.449216 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:00.448884    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:09:00.471412 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:00.466921    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r" podStartSLOduration=1.466902359 podStartE2EDuration="1.466902359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:09:00.465546038 +0000 UTC m=+2414.858758934" watchObservedRunningTime="2026-07-29 11:09:00.466902359 +0000 UTC m=+2414.860115230"
Jul 29 11:09:00.939273 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:00.939231    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:09:06.396224 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:06.396182    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:09:06.495801 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:06.495768    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:09:08.015649 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:08.015615    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:09:21.261727 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.261689    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:09:21.538588 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.538506    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"]
Jul 29 11:09:21.542141 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.542084    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:09:21.545187 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.545157    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"]
Jul 29 11:09:21.546527 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.546501    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-rqgbm\""
Jul 29 11:09:21.546677 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.546605    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-dq8z6\""
Jul 29 11:09:21.546677 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.546658    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 29 11:09:21.546677 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.546667    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 29 11:09:21.546844 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.546685    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 29 11:09:21.554712 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.554681    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:09:21.611683 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.611645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjk6g\" (UniqueName: \"kubernetes.io/projected/3f9406bf-5a4f-4868-9afc-4770777d5e75-kube-api-access-mjk6g\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms\" (UID: \"3f9406bf-5a4f-4868-9afc-4770777d5e75\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:09:21.611892 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.611814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9406bf-5a4f-4868-9afc-4770777d5e75-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms\" (UID: \"3f9406bf-5a4f-4868-9afc-4770777d5e75\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:09:21.712540 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.712506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjk6g\" (UniqueName: \"kubernetes.io/projected/3f9406bf-5a4f-4868-9afc-4770777d5e75-kube-api-access-mjk6g\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms\" (UID: \"3f9406bf-5a4f-4868-9afc-4770777d5e75\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:09:21.712729 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.712549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9406bf-5a4f-4868-9afc-4770777d5e75-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms\" (UID: \"3f9406bf-5a4f-4868-9afc-4770777d5e75\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:09:21.715171 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.715145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9406bf-5a4f-4868-9afc-4770777d5e75-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms\" (UID: \"3f9406bf-5a4f-4868-9afc-4770777d5e75\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:09:21.720939 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.720917    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjk6g\" (UniqueName: \"kubernetes.io/projected/3f9406bf-5a4f-4868-9afc-4770777d5e75-kube-api-access-mjk6g\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms\" (UID: \"3f9406bf-5a4f-4868-9afc-4770777d5e75\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:09:21.855928 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.855896    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:09:21.994358 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:21.994324    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"]
Jul 29 11:09:21.997768 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:09:21.997739    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9406bf_5a4f_4868_9afc_4770777d5e75.slice/crio-71b5b36a44181b0642aac99022ff674dcb4b50ed56b329ab536c7ab143fddbae WatchSource:0}: Error finding container 71b5b36a44181b0642aac99022ff674dcb4b50ed56b329ab536c7ab143fddbae: Status 404 returned error can't find the container with id 71b5b36a44181b0642aac99022ff674dcb4b50ed56b329ab536c7ab143fddbae
Jul 29 11:09:22.560648 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:22.560613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms" event={"ID":"3f9406bf-5a4f-4868-9afc-4770777d5e75","Type":"ContainerStarted","Data":"464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e"}
Jul 29 11:09:22.561066 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:22.560666    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms" event={"ID":"3f9406bf-5a4f-4868-9afc-4770777d5e75","Type":"ContainerStarted","Data":"71b5b36a44181b0642aac99022ff674dcb4b50ed56b329ab536c7ab143fddbae"}
Jul 29 11:09:22.561066 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:22.560697    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:09:22.576684 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:22.576628    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms" podStartSLOduration=1.5766117259999999 podStartE2EDuration="1.576611726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:09:22.576234794 +0000 UTC m=+2436.969447664" watchObservedRunningTime="2026-07-29 11:09:22.576611726 +0000 UTC m=+2436.969824596"
Jul 29 11:09:31.455930 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:31.455895    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:09:34.922894 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:34.922862    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:09:53.567727 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:09:53.567693    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:10:53.263626 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:10:53.263588    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:21.116577 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:21.116542    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"]
Jul 29 11:11:21.120590 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:21.120562    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:21.339036 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:21.338995    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:23.567704 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:23.567669    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms" podUID="3f9406bf-5a4f-4868-9afc-4770777d5e75" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:11:33.567828 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:33.567786    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms" podUID="3f9406bf-5a4f-4868-9afc-4770777d5e75" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:11:36.139155 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:36.139086    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms" podUID="3f9406bf-5a4f-4868-9afc-4770777d5e75" containerName="main" containerID="cri-o://464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e" gracePeriod=45
Jul 29 11:11:36.387417 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:36.387389    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:11:36.458023 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:36.457939    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9406bf-5a4f-4868-9afc-4770777d5e75-tls-certs\") pod \"3f9406bf-5a4f-4868-9afc-4770777d5e75\" (UID: \"3f9406bf-5a4f-4868-9afc-4770777d5e75\") "
Jul 29 11:11:36.458180 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:36.458047    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjk6g\" (UniqueName: \"kubernetes.io/projected/3f9406bf-5a4f-4868-9afc-4770777d5e75-kube-api-access-mjk6g\") pod \"3f9406bf-5a4f-4868-9afc-4770777d5e75\" (UID: \"3f9406bf-5a4f-4868-9afc-4770777d5e75\") "
Jul 29 11:11:36.460289 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:36.460252    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9406bf-5a4f-4868-9afc-4770777d5e75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f9406bf-5a4f-4868-9afc-4770777d5e75" (UID: "3f9406bf-5a4f-4868-9afc-4770777d5e75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:11:36.460412 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:36.460378    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9406bf-5a4f-4868-9afc-4770777d5e75-kube-api-access-mjk6g" (OuterVolumeSpecName: "kube-api-access-mjk6g") pod "3f9406bf-5a4f-4868-9afc-4770777d5e75" (UID: "3f9406bf-5a4f-4868-9afc-4770777d5e75"). InnerVolumeSpecName "kube-api-access-mjk6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:11:36.559667 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:36.559625    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9406bf-5a4f-4868-9afc-4770777d5e75-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:11:36.559667 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:36.559661    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjk6g\" (UniqueName: \"kubernetes.io/projected/3f9406bf-5a4f-4868-9afc-4770777d5e75-kube-api-access-mjk6g\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:11:37.201644 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:37.201610    2569 generic.go:356] "Generic (PLEG): container finished" podID="3f9406bf-5a4f-4868-9afc-4770777d5e75" containerID="464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e" exitCode=0
Jul 29 11:11:37.201644 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:37.201648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms" event={"ID":"3f9406bf-5a4f-4868-9afc-4770777d5e75","Type":"ContainerDied","Data":"464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e"}
Jul 29 11:11:37.202170 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:37.201687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms" event={"ID":"3f9406bf-5a4f-4868-9afc-4770777d5e75","Type":"ContainerDied","Data":"71b5b36a44181b0642aac99022ff674dcb4b50ed56b329ab536c7ab143fddbae"}
Jul 29 11:11:37.202170 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:37.201709    2569 scope.go:117] "RemoveContainer" containerID="464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e"
Jul 29 11:11:37.202170 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:37.201651    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"
Jul 29 11:11:37.211697 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:37.211674    2569 scope.go:117] "RemoveContainer" containerID="464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e"
Jul 29 11:11:37.212010 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:11:37.211991    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e\": container with ID starting with 464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e not found: ID does not exist" containerID="464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e"
Jul 29 11:11:37.212066 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:37.212020    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e"} err="failed to get container status \"464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e\": rpc error: code = NotFound desc = could not find container \"464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e\": container with ID starting with 464ceaa05164ff372ed879e99f4717bf4286cb8f37f1a7e6398e2dcf7909a92e not found: ID does not exist"
Jul 29 11:11:37.223261 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:37.223228    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"]
Jul 29 11:11:37.226933 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:37.226906    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejdhms"]
Jul 29 11:11:38.042345 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:38.042312    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f9406bf-5a4f-4868-9afc-4770777d5e75" path="/var/lib/kubelet/pods/3f9406bf-5a4f-4868-9afc-4770777d5e75/volumes"
Jul 29 11:11:41.098104 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:41.098027    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:41.198064 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:41.198030    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:41.298742 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:41.298707    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:41.398998 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:41.398906    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:41.499636 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:41.499601    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:43.621085 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:43.621046    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:43.718791 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:43.718756    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:44.394661 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:44.394624    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:46.177669 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:46.177635    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:46.276318 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:46.276287    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:47.175069 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:47.175033    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:52.718098 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:52.718065    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:52.781857 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:52.781817    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:53.545907 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:53.545875    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:11:53.647363 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:11:53.647332    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:12:03.876536 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.876494    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"]
Jul 29 11:12:03.877019 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.876888    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9406bf-5a4f-4868-9afc-4770777d5e75" containerName="main"
Jul 29 11:12:03.877019 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.876900    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9406bf-5a4f-4868-9afc-4770777d5e75" containerName="main"
Jul 29 11:12:03.877019 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.876975    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9406bf-5a4f-4868-9afc-4770777d5e75" containerName="main"
Jul 29 11:12:03.880238 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.880214    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:03.883595 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.883140    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 29 11:12:03.883595 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.883257    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 29 11:12:03.883595 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.883410    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 29 11:12:03.883842 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.883628    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-mcnx5\""
Jul 29 11:12:03.892343 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.892315    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"]
Jul 29 11:12:03.918094 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.918060    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:12:03.995163 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.995129    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttlhz\" (UniqueName: \"kubernetes.io/projected/db0a398f-6814-400e-bb81-fcae5bc445af-kube-api-access-ttlhz\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:03.995360 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.995175    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:03.995360 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.995197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0a398f-6814-400e-bb81-fcae5bc445af-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:03.995360 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.995310    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:03.995538 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.995364    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:03.995538 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:03.995387    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.096755 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.096723    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.096755 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.096756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.097015 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.096802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttlhz\" (UniqueName: \"kubernetes.io/projected/db0a398f-6814-400e-bb81-fcae5bc445af-kube-api-access-ttlhz\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.097015 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.096848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.097015 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.096876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0a398f-6814-400e-bb81-fcae5bc445af-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.097015 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.096998    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.097228 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.097184    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.097294 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.097268    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.097341 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.097326    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.099127 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.099104    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0a398f-6814-400e-bb81-fcae5bc445af-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.099409 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.099391    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0a398f-6814-400e-bb81-fcae5bc445af-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.109364 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.109342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttlhz\" (UniqueName: \"kubernetes.io/projected/db0a398f-6814-400e-bb81-fcae5bc445af-kube-api-access-ttlhz\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk\" (UID: \"db0a398f-6814-400e-bb81-fcae5bc445af\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:12:04.120858 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:04.120828    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:12:13.274765 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:13.274680    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:12:34.544869 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:12:34.544826    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:14:06.892710 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:14:06.892655    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:14:06.912834 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:14:06.912802    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:14:13.399103 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:14:13.399064    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:14:13.492685 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:14:13.492648    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:14:46.044833 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:14:46.044801    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:14:46.145317 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:14:46.145282    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:14:54.325882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:14:54.325846    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:16:09.914154 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:16:09.914105    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:16:10.496914 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:16:10.496880    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:18:12.331906 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:18:12.331825    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:18:13.497750 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:18:13.497703    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:18:14.066919 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:18:14.066884    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:19:06.171592 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:19:06.171558    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:20:17.068239 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:20:17.068192    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:20:17.666056 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:20:17.666019    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:22:20.666805 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:22:20.666757    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:22:21.259087 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:22:21.259053    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:23:59.994345 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:23:59.994310    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"]
Jul 29 11:24:01.434851 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:01.434816    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:24:01.456022 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:01.455988    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r" podUID="b96d50e7-17a7-4a76-97a2-a046205a13f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:24:02.849083 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:02.849029    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:24:11.455880 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:11.455836    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r" podUID="b96d50e7-17a7-4a76-97a2-a046205a13f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:24:15.015311 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.015264    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r" podUID="b96d50e7-17a7-4a76-97a2-a046205a13f3" containerName="main" containerID="cri-o://757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db" gracePeriod=45
Jul 29 11:24:15.274254 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.274189    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:24:15.381688 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.381642    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d50e7-17a7-4a76-97a2-a046205a13f3-tls-certs\") pod \"b96d50e7-17a7-4a76-97a2-a046205a13f3\" (UID: \"b96d50e7-17a7-4a76-97a2-a046205a13f3\") "
Jul 29 11:24:15.381895 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.381713    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/b96d50e7-17a7-4a76-97a2-a046205a13f3-kube-api-access-f6hrl\") pod \"b96d50e7-17a7-4a76-97a2-a046205a13f3\" (UID: \"b96d50e7-17a7-4a76-97a2-a046205a13f3\") "
Jul 29 11:24:15.383924 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.383901    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96d50e7-17a7-4a76-97a2-a046205a13f3-kube-api-access-f6hrl" (OuterVolumeSpecName: "kube-api-access-f6hrl") pod "b96d50e7-17a7-4a76-97a2-a046205a13f3" (UID: "b96d50e7-17a7-4a76-97a2-a046205a13f3"). InnerVolumeSpecName "kube-api-access-f6hrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:24:15.384020 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.383902    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96d50e7-17a7-4a76-97a2-a046205a13f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b96d50e7-17a7-4a76-97a2-a046205a13f3" (UID: "b96d50e7-17a7-4a76-97a2-a046205a13f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:24:15.483388 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.483348    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d50e7-17a7-4a76-97a2-a046205a13f3-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:24:15.483388 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.483382    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hrl\" (UniqueName: \"kubernetes.io/projected/b96d50e7-17a7-4a76-97a2-a046205a13f3-kube-api-access-f6hrl\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:24:15.806115 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.806081    2569 generic.go:356] "Generic (PLEG): container finished" podID="b96d50e7-17a7-4a76-97a2-a046205a13f3" containerID="757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db" exitCode=0
Jul 29 11:24:15.806326 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.806148    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"
Jul 29 11:24:15.806326 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.806174    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r" event={"ID":"b96d50e7-17a7-4a76-97a2-a046205a13f3","Type":"ContainerDied","Data":"757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db"}
Jul 29 11:24:15.806326 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.806225    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r" event={"ID":"b96d50e7-17a7-4a76-97a2-a046205a13f3","Type":"ContainerDied","Data":"9a11edc39a39f822885015dffb8dbe1317ae30c1b555a1627922bc4fb4b70463"}
Jul 29 11:24:15.806326 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.806248    2569 scope.go:117] "RemoveContainer" containerID="757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db"
Jul 29 11:24:15.815796 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.815773    2569 scope.go:117] "RemoveContainer" containerID="757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db"
Jul 29 11:24:15.816039 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:24:15.816017    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db\": container with ID starting with 757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db not found: ID does not exist" containerID="757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db"
Jul 29 11:24:15.816139 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.816044    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db"} err="failed to get container status \"757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db\": rpc error: code = NotFound desc = could not find container \"757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db\": container with ID starting with 757c81a7a4e8ca98a4a648a3224452df5d99e58056d1e9f8ad1529faed1f90db not found: ID does not exist"
Jul 29 11:24:15.828143 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.828117    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"]
Jul 29 11:24:15.829769 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:15.829751    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-hw94r"]
Jul 29 11:24:16.046911 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:16.046877    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96d50e7-17a7-4a76-97a2-a046205a13f3" path="/var/lib/kubelet/pods/b96d50e7-17a7-4a76-97a2-a046205a13f3/volumes"
Jul 29 11:24:20.022259 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:20.022224    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:24:20.122766 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:20.122728    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:24:24.259956 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:24:24.259908    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:24:24.848177 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:24.848143    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:24:35.959387 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:35.959351    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:24:36.048175 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.048142    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:24:36.117297 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.117263    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:24:36.221377 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.221301    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"]
Jul 29 11:24:36.221697 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.221679    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96d50e7-17a7-4a76-97a2-a046205a13f3" containerName="main"
Jul 29 11:24:36.221697 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.221695    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96d50e7-17a7-4a76-97a2-a046205a13f3" containerName="main"
Jul 29 11:24:36.221864 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.221770    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96d50e7-17a7-4a76-97a2-a046205a13f3" containerName="main"
Jul 29 11:24:36.224744 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.224726    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:24:36.229492 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.229467    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 29 11:24:36.229607 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.229546    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 29 11:24:36.229607 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.229570    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-29cvz\""
Jul 29 11:24:36.229777 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.229748    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-epp-sa-dockercfg-hdk7w\""
Jul 29 11:24:36.229777 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.229747    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 29 11:24:36.237385 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.237360    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"]
Jul 29 11:24:36.255445 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.255403    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:24:36.373327 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.373283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h\" (UID: \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:24:36.373571 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.373498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsp5\" (UniqueName: \"kubernetes.io/projected/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-kube-api-access-pnsp5\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h\" (UID: \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:24:36.474852 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.474754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsp5\" (UniqueName: \"kubernetes.io/projected/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-kube-api-access-pnsp5\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h\" (UID: \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:24:36.474852 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.474828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h\" (UID: \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:24:36.477388 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.477359    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h\" (UID: \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:24:36.482200 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.482173    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsp5\" (UniqueName: \"kubernetes.io/projected/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-kube-api-access-pnsp5\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h\" (UID: \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:24:36.536096 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.536062    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:24:36.677468 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.677443    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"]
Jul 29 11:24:36.679960 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:24:36.679932    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49fabd4_ee62_49ec_9a8b_9ab0f65dd4e6.slice/crio-f2a7d31d4be2b260269d5a9aa2ba1d1466abcdf5a5ab08f0d03a0d0e10450ee2 WatchSource:0}: Error finding container f2a7d31d4be2b260269d5a9aa2ba1d1466abcdf5a5ab08f0d03a0d0e10450ee2: Status 404 returned error can't find the container with id f2a7d31d4be2b260269d5a9aa2ba1d1466abcdf5a5ab08f0d03a0d0e10450ee2
Jul 29 11:24:36.911924 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.911884    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h" event={"ID":"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6","Type":"ContainerStarted","Data":"087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170"}
Jul 29 11:24:36.912117 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.911945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h" event={"ID":"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6","Type":"ContainerStarted","Data":"f2a7d31d4be2b260269d5a9aa2ba1d1466abcdf5a5ab08f0d03a0d0e10450ee2"}
Jul 29 11:24:36.912117 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.912071    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:24:36.934291 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:36.934237    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h" podStartSLOduration=0.934218198 podStartE2EDuration="934.218198ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:24:36.932700738 +0000 UTC m=+3351.325913629" watchObservedRunningTime="2026-07-29 11:24:36.934218198 +0000 UTC m=+3351.327431069"
Jul 29 11:24:52.298219 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:24:52.298178    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:25:02.786857 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:25:02.786821    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:25:07.919011 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:25:07.918979    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:26:27.849565 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:26:27.849516    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:26:28.444779 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:26:28.444746    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:26:50.658655 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:26:50.658615    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:27:04.129447 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:27:04.129392    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:27:04.231043 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:27:04.231008    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:27:04.330484 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:27:04.330450    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:27:04.429380 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:27:04.429298    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:27:06.575346 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:27:06.575314    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:27:06.676136 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:27:06.676107    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:27:07.398638 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:27:07.398600    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:28:06.655536 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.655500    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:28:06.741206 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.741171    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:28:06.791491 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.791459    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 29 11:28:06.794825 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.794808    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:06.798743 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.798718    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-brhl9\""
Jul 29 11:28:06.798882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.798719    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-5lcx4\""
Jul 29 11:28:06.798882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.798766    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 29 11:28:06.798882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.798772    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 29 11:28:06.798882 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.798719    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 29 11:28:06.803877 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.803853    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 29 11:28:06.814124 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.814100    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:28:06.919388 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.919306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:06.919388 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.919335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:06.919627 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.919414    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:06.919627 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.919519    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7f3375-271c-4793-ba1f-89b23a56fb52-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:06.919627 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.919553    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z568m\" (UniqueName: \"kubernetes.io/projected/0b7f3375-271c-4793-ba1f-89b23a56fb52-kube-api-access-z568m\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:06.919627 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.919622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:06.945751 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:06.945718    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:28:07.020094 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.020061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.020094 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.020099    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.020356 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.020123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.020356 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.020189    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.020356 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.020263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7f3375-271c-4793-ba1f-89b23a56fb52-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.020356 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.020293    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z568m\" (UniqueName: \"kubernetes.io/projected/0b7f3375-271c-4793-ba1f-89b23a56fb52-kube-api-access-z568m\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.020570 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.020469    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.020720 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.020701    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.020830 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.020808    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.022888 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.022855    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b7f3375-271c-4793-ba1f-89b23a56fb52-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.023169 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.023148    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7f3375-271c-4793-ba1f-89b23a56fb52-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:07.027917 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:07.027895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z568m\" (UniqueName: \"kubernetes.io/projected/0b7f3375-271c-4793-ba1f-89b23a56fb52-kube-api-access-z568m\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:09.746144 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:09.746103    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:09.753940 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:09.753890    2569 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:09.847281 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:09.847243    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:09.847511 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:09.847403    2569 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:09.850301 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:09.850275    2569 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") DevicePath \"csi-073bc2b8f16dfea97aee88d62910a618d9c2b6da144d4a74033e2cdbcfbe1d32\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:09.946387 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:09.946348    2569 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/82708568a91e8258bb13db9f7d1800f3b22eae7de45eb7c99da7ab5491d26471/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:09.959505 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:09.959467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b8b0e167-4bf3-4d8d-86ab-b583ee0ba857\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-056cfc3ce0d837221\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0b7f3375-271c-4793-ba1f-89b23a56fb52\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:10.106396 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:10.106348    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 11:28:10.245966 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:10.245934    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 29 11:28:10.248109 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:28:10.248079    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7f3375_271c_4793_ba1f_89b23a56fb52.slice/crio-2d975af3d69075fae9f1e01512db31f9eaae76883d3d11116b152fd0c26e0e47 WatchSource:0}: Error finding container 2d975af3d69075fae9f1e01512db31f9eaae76883d3d11116b152fd0c26e0e47: Status 404 returned error can't find the container with id 2d975af3d69075fae9f1e01512db31f9eaae76883d3d11116b152fd0c26e0e47
Jul 29 11:28:10.250838 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:10.250814    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 11:28:10.949356 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:10.949256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"0b7f3375-271c-4793-ba1f-89b23a56fb52","Type":"ContainerStarted","Data":"2d975af3d69075fae9f1e01512db31f9eaae76883d3d11116b152fd0c26e0e47"}
Jul 29 11:28:31.266505 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:31.266458    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:28:31.367151 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:31.367096    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:28:31.445481 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:28:31.445404    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:28:32.082600 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:32.082561    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:28:39.128902 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:39.128809    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"0b7f3375-271c-4793-ba1f-89b23a56fb52","Type":"ContainerStarted","Data":"21be8d3388a3ccc86a6cd6eea666f608afb8836736640ad9fd5bb73a1cbf43a8"}
Jul 29 11:28:39.148250 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:39.148195    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=4.562910063 podStartE2EDuration="33.148178314s" totalImagesPullingTime="28.585268251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 11:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:28:39.146212821 +0000 UTC m=+3593.539425692" watchObservedRunningTime="2026-07-29 11:28:39.148178314 +0000 UTC m=+3593.541391186"
Jul 29 11:28:43.621368 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:28:43.621321    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:29:00.143685 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:29:00.143645    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:29:48.792575 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:29:48.792532    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:30:35.083697 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:30:35.083652    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:30:35.723221 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:30:35.723181    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:32:38.724172 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:32:38.724126    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:32:39.314576 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:32:39.314536    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:33:00.664030 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:33:00.663990    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:34:42.316197 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:34:42.316103    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:34:42.906579 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:34:42.906542    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:36:45.907064 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:36:45.907012    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:36:46.518800 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:36:46.518768    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:38:26.328336 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:38:26.328294    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:38:29.644823 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:38:29.644783    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:38:49.519407 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:38:49.519364    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:38:50.114413 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:38:50.114380    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:39:54.658477 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:39:54.658436    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:39:54.664810 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:39:54.664775    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"]
Jul 29 11:39:54.756276 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:39:54.756242    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:39:57.918520 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:39:57.918474    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h" podUID="d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:40:07.918848 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:07.918809    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h" podUID="d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:40:09.684458 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:09.684395    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h" podUID="d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" containerName="main" containerID="cri-o://087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170" gracePeriod=45
Jul 29 11:40:09.938708 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:09.938641    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:40:09.980672 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:09.980630    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnsp5\" (UniqueName: \"kubernetes.io/projected/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-kube-api-access-pnsp5\") pod \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\" (UID: \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\") "
Jul 29 11:40:09.982833 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:09.982803    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-kube-api-access-pnsp5" (OuterVolumeSpecName: "kube-api-access-pnsp5") pod "d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" (UID: "d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6"). InnerVolumeSpecName "kube-api-access-pnsp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:40:10.081394 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.081358    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-tls-certs\") pod \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\" (UID: \"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6\") "
Jul 29 11:40:10.081588 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.081564    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnsp5\" (UniqueName: \"kubernetes.io/projected/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-kube-api-access-pnsp5\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:40:10.083450 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.083412    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" (UID: "d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:40:10.182523 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.182490    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:40:10.504604 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.504571    2569 generic.go:356] "Generic (PLEG): container finished" podID="d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" containerID="087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170" exitCode=0
Jul 29 11:40:10.504785 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.504618    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"
Jul 29 11:40:10.504785 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.504636    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h" event={"ID":"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6","Type":"ContainerDied","Data":"087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170"}
Jul 29 11:40:10.504785 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.504676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h" event={"ID":"d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6","Type":"ContainerDied","Data":"f2a7d31d4be2b260269d5a9aa2ba1d1466abcdf5a5ab08f0d03a0d0e10450ee2"}
Jul 29 11:40:10.504785 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.504700    2569 scope.go:117] "RemoveContainer" containerID="087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170"
Jul 29 11:40:10.513526 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.513503    2569 scope.go:117] "RemoveContainer" containerID="087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170"
Jul 29 11:40:10.513785 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:40:10.513763    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170\": container with ID starting with 087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170 not found: ID does not exist" containerID="087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170"
Jul 29 11:40:10.513841 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.513795    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170"} err="failed to get container status \"087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170\": rpc error: code = NotFound desc = could not find container \"087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170\": container with ID starting with 087ebbff37119775e3a70ad24b145c22ac5b22d415c8ac2c79acb80fbb338170 not found: ID does not exist"
Jul 29 11:40:10.526544 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.526505    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"]
Jul 29 11:40:10.528693 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:10.528671    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-4fh8h"]
Jul 29 11:40:12.042158 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:12.042121    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" path="/var/lib/kubelet/pods/d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6/volumes"
Jul 29 11:40:12.127340 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:12.127305    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:40:12.227861 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:12.227828    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:40:33.930846 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:33.930805    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:40:34.029958 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.029920    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:40:34.151251 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.151187    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"]
Jul 29 11:40:34.151812 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.151776    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" containerName="main"
Jul 29 11:40:34.151812 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.151802    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" containerName="main"
Jul 29 11:40:34.152003 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.151886    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49fabd4-ee62-49ec-9a8b-9ab0f65dd4e6" containerName="main"
Jul 29 11:40:34.156789 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.156766    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:40:34.159502 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.159477    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 29 11:40:34.160629 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.160604    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-epp-sa-dockercfg-trkfs\""
Jul 29 11:40:34.160792 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.160770    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 29 11:40:34.160893 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.160873    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 29 11:40:34.161002 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.160988    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-gp96d\""
Jul 29 11:40:34.167517 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.167488    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"]
Jul 29 11:40:34.256764 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.256680    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:40:34.305902 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.305863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8p7\" (UniqueName: \"kubernetes.io/projected/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-kube-api-access-hz8p7\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx\" (UID: \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:40:34.306089 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.305965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx\" (UID: \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:40:34.406539 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.406501    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx\" (UID: \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:40:34.406729 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.406595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8p7\" (UniqueName: \"kubernetes.io/projected/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-kube-api-access-hz8p7\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx\" (UID: \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:40:34.409183 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.409155    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx\" (UID: \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:40:34.414642 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.414611    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8p7\" (UniqueName: \"kubernetes.io/projected/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-kube-api-access-hz8p7\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx\" (UID: \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:40:34.469273 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.469239    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:40:34.609745 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.609715    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"]
Jul 29 11:40:34.611724 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:40:34.611690    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba89c8e_1dc1_46ba_84dc_5fa6f8c90b55.slice/crio-f6e08a75474fa4fd521943cad526f164ccc8b9c261a3a7083351efa6ec26b186 WatchSource:0}: Error finding container f6e08a75474fa4fd521943cad526f164ccc8b9c261a3a7083351efa6ec26b186: Status 404 returned error can't find the container with id f6e08a75474fa4fd521943cad526f164ccc8b9c261a3a7083351efa6ec26b186
Jul 29 11:40:34.628720 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:34.628689    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx" event={"ID":"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55","Type":"ContainerStarted","Data":"f6e08a75474fa4fd521943cad526f164ccc8b9c261a3a7083351efa6ec26b186"}
Jul 29 11:40:35.635842 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:35.635803    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx" event={"ID":"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55","Type":"ContainerStarted","Data":"0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073"}
Jul 29 11:40:35.636252 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:35.635921    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:40:35.652563 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:35.652504    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx" podStartSLOduration=1.652486352 podStartE2EDuration="1.652486352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:40:35.650153411 +0000 UTC m=+4310.043366286" watchObservedRunningTime="2026-07-29 11:40:35.652486352 +0000 UTC m=+4310.045699237"
Jul 29 11:40:53.115027 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:40:53.114980    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:40:53.729716 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:40:53.729682    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:41:06.644041 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:41:06.644007    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:41:37.450649 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:41:37.450613    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:17.580288 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:17.580200    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:26.106511 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:26.106472    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7c3633d1_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_0b7f3375-271c-4793-ba1f-89b23a56fb52/main/0.log"
Jul 29 11:42:27.239013 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:27.238979    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:27.341595 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:27.341561    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:29.454866 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:29.454812    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:29.558641 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:29.558589    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:29.656510 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:29.656480    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:35.545220 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:35.545167    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:35.608615 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:35.608585    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:36.536243 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:36.536208    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:36.636494 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:36.636457    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:36.736195 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:36.736158    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:40.766303 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:40.766265    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:40.867101 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:40.867063    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:40.990607 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:40.990571    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:42.175442 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:42.175391    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:42.275057 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:42.275022    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:45.529167 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:45.529133    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:45.849599 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:45.849570    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:45.910970 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:45.910939    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:46.909071 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:46.909026    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:47.060452 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:47.060392    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:50.832681 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:50.832651    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:50.933188 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:50.933152    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:51.038193 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:51.038161    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:51.604033 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:51.603999    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:51.663312 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:51.663279    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:51.931931 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:51.931847    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:52.034978 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:52.034943    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:52.170144 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:52.170111    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:52.746936 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:52.746902    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:52.847082 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:52.847052    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:53.070596 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:53.070514    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:53.732405 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:53.732369    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:53.833114 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:53.833080    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:53.932653 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:53.932614    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:54.446590 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:54.446554    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:54.549479 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:54.549447    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:54.647094 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:54.647060    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:55.328831 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:55.328795    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:55.429781 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:55.429745    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:56.689381 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:56.689345    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:56.730396 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:42:56.730354    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:42:56.791054 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:56.791019    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:57.327573 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:57.327540    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:42:57.880641 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:57.880604    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:57.980184 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:57.980146    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:58.715644 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:58.715611    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:58.815340 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:58.815308    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:59.522926 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:59.522893    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:42:59.581755 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:42:59.581725    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:43:00.294333 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:43:00.294295    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:43:00.394723 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:43:00.394685    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:43:01.116486 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:43:01.116446    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:43:01.219064 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:43:01.219031    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:43:39.556550 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:43:39.556314    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:43:41.091886 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:43:41.091844    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:44:09.661315 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:44:09.661275    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:45:00.328234 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:45:00.328180    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:45:00.922558 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:45:00.922517    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:45:36.650835 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:45:36.650733    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:45:56.260779 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:45:56.260742    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:46:06.031382 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:46:06.031343    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:46:07.037196 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:46:07.037163    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:46:07.137288 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:46:07.137255    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:46:22.634330 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:46:22.634294    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:46:22.793135 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:46:22.793106    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:46:35.458822 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:46:35.458783    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:46:35.559598 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:46:35.559562    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:46:45.451453 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:46:45.451390    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:47:03.924153 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:47:03.924107    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:47:04.517966 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:47:04.517924    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:48:28.013723 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:48:28.013643    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:49:07.519158 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:49:07.519109    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:49:08.113818 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:49:08.113784    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:49:22.421822 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:49:22.421785    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:49:24.370060 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:49:24.370011    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:49:41.252167 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:49:41.252079    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:49:41.345054 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:49:41.345021    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:49:47.303196 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:49:47.303156    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:49:52.803071 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:49:52.803032    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:49:52.903891 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:49:52.903853    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:50:07.725748 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:50:07.725714    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:51:11.115336 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:51:11.115244    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:51:11.726504 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:51:11.726470    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:51:58.541832 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:51:58.541763    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:51:58.630998 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:51:58.630959    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:53:14.727394 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:53:14.727342    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:53:15.330797 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:53:15.330762    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:53:16.393019 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:53:16.392981    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:53:33.249445 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:53:33.249406    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:53:33.352103 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:53:33.352068    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:53:33.453441 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:53:33.453393    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:53:38.931568 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:53:38.931528    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:53:38.989904 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:53:38.989849    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:53:45.567961 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:53:45.567923    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:55:13.696582 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:13.696545    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:55:18.331861 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:55:18.331806    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:55:18.929234 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:18.929202    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:55:24.050914 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:24.050844    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:55:24.059280 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:24.059231    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"]
Jul 29 11:55:24.113230 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:24.113182    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:55:26.644190 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:26.644094    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx" podUID="7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:55:36.644739 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:36.644695    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx" podUID="7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:55:39.081305 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:39.081183    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx" podUID="7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" containerName="main" containerID="cri-o://0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073" gracePeriod=45
Jul 29 11:55:39.342052 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:39.341992    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:55:39.362780 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:39.362749    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-tls-certs\") pod \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\" (UID: \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\") "
Jul 29 11:55:39.362955 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:39.362844    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz8p7\" (UniqueName: \"kubernetes.io/projected/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-kube-api-access-hz8p7\") pod \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\" (UID: \"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55\") "
Jul 29 11:55:39.365048 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:39.365012    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" (UID: "7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:55:39.365181 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:39.365130    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-kube-api-access-hz8p7" (OuterVolumeSpecName: "kube-api-access-hz8p7") pod "7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" (UID: "7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55"). InnerVolumeSpecName "kube-api-access-hz8p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:55:39.464153 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:39.464116    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz8p7\" (UniqueName: \"kubernetes.io/projected/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-kube-api-access-hz8p7\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:55:39.464153 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:39.464145    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:55:40.031277 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:40.031235    2569 generic.go:356] "Generic (PLEG): container finished" podID="7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" containerID="0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073" exitCode=0
Jul 29 11:55:40.031507 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:40.031288    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"
Jul 29 11:55:40.031507 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:40.031307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx" event={"ID":"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55","Type":"ContainerDied","Data":"0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073"}
Jul 29 11:55:40.031507 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:40.031353    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx" event={"ID":"7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55","Type":"ContainerDied","Data":"f6e08a75474fa4fd521943cad526f164ccc8b9c261a3a7083351efa6ec26b186"}
Jul 29 11:55:40.031507 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:40.031383    2569 scope.go:117] "RemoveContainer" containerID="0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073"
Jul 29 11:55:40.046448 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:40.046406    2569 scope.go:117] "RemoveContainer" containerID="0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073"
Jul 29 11:55:40.046762 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:55:40.046734    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073\": container with ID starting with 0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073 not found: ID does not exist" containerID="0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073"
Jul 29 11:55:40.046821 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:40.046778    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073"} err="failed to get container status \"0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073\": rpc error: code = NotFound desc = could not find container \"0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073\": container with ID starting with 0c5be0d6451f8a5ad8cf4a32c9c997f95e5fb64e0d7057a87468a3e19a864073 not found: ID does not exist"
Jul 29 11:55:40.055124 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:40.055094    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"]
Jul 29 11:55:40.061112 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:40.061087    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-4cljx"]
Jul 29 11:55:42.042176 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:42.042134    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" path="/var/lib/kubelet/pods/7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55/volumes"
Jul 29 11:55:44.330480 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:44.330445    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:55:44.433832 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:44.433779    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:55:57.054126 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:57.054090    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:55:57.269931 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:55:57.269858    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:56:03.935055 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:56:03.935004    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:56:04.017450 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:56:04.017396    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:56:16.465841 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:56:16.465793    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:56:31.430977 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:56:31.430941    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:56:35.137245 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:56:35.137209    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:56:35.237557 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:56:35.237515    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:56:35.341049 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:56:35.341008    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:56:40.929004 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:56:40.928967    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:56:41.029651 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:56:41.029618    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:57:17.344454 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.344355    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:57:17.517834 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.517796    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"]
Jul 29 11:57:17.518667 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.518638    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" containerName="main"
Jul 29 11:57:17.518667 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.518666    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" containerName="main"
Jul 29 11:57:17.518821 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.518774    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba89c8e-1dc1-46ba-84dc-5fa6f8c90b55" containerName="main"
Jul 29 11:57:17.522128 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.522107    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:57:17.526189 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.526169    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-6dsrf\""
Jul 29 11:57:17.526307 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.526172    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 29 11:57:17.526307 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.526296    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-x6gvf\""
Jul 29 11:57:17.526448 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.526175    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 29 11:57:17.526448 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.526208    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 29 11:57:17.529955 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.529930    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"]
Jul 29 11:57:17.546112 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.546088    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:57:17.587336 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.587295    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gqw\" (UniqueName: \"kubernetes.io/projected/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-kube-api-access-j8gqw\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-v7hgt\" (UID: \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:57:17.587548 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.587366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-v7hgt\" (UID: \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:57:17.688494 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.688372    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gqw\" (UniqueName: \"kubernetes.io/projected/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-kube-api-access-j8gqw\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-v7hgt\" (UID: \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:57:17.688494 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.688484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-v7hgt\" (UID: \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:57:17.691100 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.691070    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-v7hgt\" (UID: \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:57:17.696482 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.696457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gqw\" (UniqueName: \"kubernetes.io/projected/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-kube-api-access-j8gqw\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-v7hgt\" (UID: \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:57:17.834153 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.834119    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:57:17.977621 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:17.977598    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"]
Jul 29 11:57:17.979854 ip-10-0-129-80 kubenswrapper[2569]: W0729 11:57:17.979814    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e15ae41_cba4_4ffd_b2c7_3ac440dc66bf.slice/crio-15f62f4caf2d220f36db40039753391b03eaf2e28b72b29b5799f41e01663c2d WatchSource:0}: Error finding container 15f62f4caf2d220f36db40039753391b03eaf2e28b72b29b5799f41e01663c2d: Status 404 returned error can't find the container with id 15f62f4caf2d220f36db40039753391b03eaf2e28b72b29b5799f41e01663c2d
Jul 29 11:57:18.515780 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:18.515740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt" event={"ID":"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf","Type":"ContainerStarted","Data":"b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e"}
Jul 29 11:57:18.516198 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:18.515796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt" event={"ID":"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf","Type":"ContainerStarted","Data":"15f62f4caf2d220f36db40039753391b03eaf2e28b72b29b5799f41e01663c2d"}
Jul 29 11:57:18.516198 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:18.515871    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:57:18.532292 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:18.532241    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt" podStartSLOduration=1.532223371 podStartE2EDuration="1.532223371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:57:18.530792826 +0000 UTC m=+5312.924005703" watchObservedRunningTime="2026-07-29 11:57:18.532223371 +0000 UTC m=+5312.925436240"
Jul 29 11:57:21.930721 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:57:21.930672    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:57:22.535544 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:22.535511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:57:40.603039 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:40.603005    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:57:49.522953 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:57:49.522923    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:58:09.157406 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:58:09.157367    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:59:25.536459 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:59:25.536396    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 11:59:26.126634 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:26.126593    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 11:59:35.289781 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:35.289745    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"]
Jul 29 11:59:35.292472 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:35.292420    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:59:35.348572 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:35.348519    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:59:39.522878 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:39.522833    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt" podUID="2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:59:49.522249 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:49.522208    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt" podUID="2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 11:59:50.311041 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:50.310970    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt" podUID="2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" containerName="main" containerID="cri-o://b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e" gracePeriod=45
Jul 29 11:59:50.564419 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:50.564358    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:59:50.646412 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:50.646376    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-tls-certs\") pod \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\" (UID: \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\") "
Jul 29 11:59:50.646642 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:50.646486    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8gqw\" (UniqueName: \"kubernetes.io/projected/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-kube-api-access-j8gqw\") pod \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\" (UID: \"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf\") "
Jul 29 11:59:50.648833 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:50.648794    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-kube-api-access-j8gqw" (OuterVolumeSpecName: "kube-api-access-j8gqw") pod "2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" (UID: "2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf"). InnerVolumeSpecName "kube-api-access-j8gqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:59:50.648833 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:50.648816    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" (UID: "2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:59:50.747759 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:50.747724    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8gqw\" (UniqueName: \"kubernetes.io/projected/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-kube-api-access-j8gqw\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:59:50.747759 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:50.747756    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 11:59:51.254510 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:51.254474    2569 generic.go:356] "Generic (PLEG): container finished" podID="2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" containerID="b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e" exitCode=0
Jul 29 11:59:51.254708 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:51.254522    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"
Jul 29 11:59:51.254708 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:51.254565    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt" event={"ID":"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf","Type":"ContainerDied","Data":"b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e"}
Jul 29 11:59:51.254708 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:51.254621    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt" event={"ID":"2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf","Type":"ContainerDied","Data":"15f62f4caf2d220f36db40039753391b03eaf2e28b72b29b5799f41e01663c2d"}
Jul 29 11:59:51.254708 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:51.254646    2569 scope.go:117] "RemoveContainer" containerID="b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e"
Jul 29 11:59:51.263888 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:51.263870    2569 scope.go:117] "RemoveContainer" containerID="b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e"
Jul 29 11:59:51.264142 ip-10-0-129-80 kubenswrapper[2569]: E0729 11:59:51.264124    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e\": container with ID starting with b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e not found: ID does not exist" containerID="b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e"
Jul 29 11:59:51.264208 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:51.264149    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e"} err="failed to get container status \"b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e\": rpc error: code = NotFound desc = could not find container \"b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e\": container with ID starting with b24c90c1cb9a5fe64b13c7623e58874b893d8248d189aef6366bcb18d379e14e not found: ID does not exist"
Jul 29 11:59:51.276539 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:51.276513    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"]
Jul 29 11:59:51.278155 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:51.278135    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-v7hgt"]
Jul 29 11:59:52.041879 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:52.041846    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" path="/var/lib/kubelet/pods/2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf/volumes"
Jul 29 11:59:52.304209 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:52.304130    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:59:52.399728 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:52.399691    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 11:59:58.355774 ip-10-0-129-80 kubenswrapper[2569]: I0729 11:59:58.355737    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:00:01.011495 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:01.011462    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:00:03.962328 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:03.962284    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:00:04.011565 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:04.011527    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:00:31.975951 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:31.975912    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:00:32.184190 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.184153    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"]
Jul 29 12:00:32.184632 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.184613    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" containerName="main"
Jul 29 12:00:32.184632 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.184633    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" containerName="main"
Jul 29 12:00:32.184786 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.184744    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e15ae41-cba4-4ffd-b2c7-3ac440dc66bf" containerName="main"
Jul 29 12:00:32.188080 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.188052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:00:32.192143 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.192120    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 29 12:00:32.192143 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.192128    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-lk8cq\""
Jul 29 12:00:32.192354 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.192128    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 29 12:00:32.192354 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.192180    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-tntgg\""
Jul 29 12:00:32.192354 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.192137    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 29 12:00:32.197354 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.196933    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"]
Jul 29 12:00:32.332000 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.331965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab07f2-b993-4921-9054-a2148898824f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n\" (UID: \"a9ab07f2-b993-4921-9054-a2148898824f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:00:32.332194 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.332109    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5q9r\" (UniqueName: \"kubernetes.io/projected/a9ab07f2-b993-4921-9054-a2148898824f-kube-api-access-j5q9r\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n\" (UID: \"a9ab07f2-b993-4921-9054-a2148898824f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:00:32.433399 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.433356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5q9r\" (UniqueName: \"kubernetes.io/projected/a9ab07f2-b993-4921-9054-a2148898824f-kube-api-access-j5q9r\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n\" (UID: \"a9ab07f2-b993-4921-9054-a2148898824f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:00:32.433602 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.433419    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab07f2-b993-4921-9054-a2148898824f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n\" (UID: \"a9ab07f2-b993-4921-9054-a2148898824f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:00:32.436069 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.436044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab07f2-b993-4921-9054-a2148898824f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n\" (UID: \"a9ab07f2-b993-4921-9054-a2148898824f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:00:32.441441 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.441404    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5q9r\" (UniqueName: \"kubernetes.io/projected/a9ab07f2-b993-4921-9054-a2148898824f-kube-api-access-j5q9r\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n\" (UID: \"a9ab07f2-b993-4921-9054-a2148898824f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:00:32.501384 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.501351    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:00:32.640905 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:32.640876    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"]
Jul 29 12:00:32.643048 ip-10-0-129-80 kubenswrapper[2569]: W0729 12:00:32.643013    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ab07f2_b993_4921_9054_a2148898824f.slice/crio-8cef742198c72b3e81808c9a35dd3bb3b1b5a8f6ed3cba417430fa6460151ea2 WatchSource:0}: Error finding container 8cef742198c72b3e81808c9a35dd3bb3b1b5a8f6ed3cba417430fa6460151ea2: Status 404 returned error can't find the container with id 8cef742198c72b3e81808c9a35dd3bb3b1b5a8f6ed3cba417430fa6460151ea2
Jul 29 12:00:33.457486 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:33.457443    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n" event={"ID":"a9ab07f2-b993-4921-9054-a2148898824f","Type":"ContainerStarted","Data":"09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f"}
Jul 29 12:00:33.457947 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:33.457521    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:00:33.457947 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:33.457546    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n" event={"ID":"a9ab07f2-b993-4921-9054-a2148898824f","Type":"ContainerStarted","Data":"8cef742198c72b3e81808c9a35dd3bb3b1b5a8f6ed3cba417430fa6460151ea2"}
Jul 29 12:00:33.477535 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:33.477479    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n" podStartSLOduration=1.477459599 podStartE2EDuration="1.477459599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:00:33.47599483 +0000 UTC m=+5507.869207701" watchObservedRunningTime="2026-07-29 12:00:33.477459599 +0000 UTC m=+5507.870672468"
Jul 29 12:00:35.460082 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:00:35.460047    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:01:04.464628 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:01:04.464594    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:01:10.519681 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:01:10.519641    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:01:29.128087 ip-10-0-129-80 kubenswrapper[2569]: E0729 12:01:29.128036    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 12:01:29.743471 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:01:29.743438    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 12:02:02.539159 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:02:02.539122    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:02:02.586815 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:02:02.586778    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:02:04.074100 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:02:04.074065    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:03:22.017814 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:22.017726    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"]
Jul 29 12:03:22.023394 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:22.023339    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:03:22.079505 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:22.079460    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:03:24.464678 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:24.464638    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n" podUID="a9ab07f2-b993-4921-9054-a2148898824f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 12:03:32.744347 ip-10-0-129-80 kubenswrapper[2569]: E0729 12:03:32.744294    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 12:03:33.321933 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:33.321901    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 12:03:34.464609 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:34.464570    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n" podUID="a9ab07f2-b993-4921-9054-a2148898824f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 12:03:37.039590 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.039528    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n" podUID="a9ab07f2-b993-4921-9054-a2148898824f" containerName="main" containerID="cri-o://09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f" gracePeriod=45
Jul 29 12:03:37.308689 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.308625    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:03:37.341912 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.341874    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9ab07f2-b993-4921-9054-a2148898824f" containerID="09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f" exitCode=0
Jul 29 12:03:37.342072 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.341925    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"
Jul 29 12:03:37.342072 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.341960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n" event={"ID":"a9ab07f2-b993-4921-9054-a2148898824f","Type":"ContainerDied","Data":"09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f"}
Jul 29 12:03:37.342072 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.342024    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n" event={"ID":"a9ab07f2-b993-4921-9054-a2148898824f","Type":"ContainerDied","Data":"8cef742198c72b3e81808c9a35dd3bb3b1b5a8f6ed3cba417430fa6460151ea2"}
Jul 29 12:03:37.342072 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.342057    2569 scope.go:117] "RemoveContainer" containerID="09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f"
Jul 29 12:03:37.350537 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.350515    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5q9r\" (UniqueName: \"kubernetes.io/projected/a9ab07f2-b993-4921-9054-a2148898824f-kube-api-access-j5q9r\") pod \"a9ab07f2-b993-4921-9054-a2148898824f\" (UID: \"a9ab07f2-b993-4921-9054-a2148898824f\") "
Jul 29 12:03:37.350662 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.350597    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab07f2-b993-4921-9054-a2148898824f-tls-certs\") pod \"a9ab07f2-b993-4921-9054-a2148898824f\" (UID: \"a9ab07f2-b993-4921-9054-a2148898824f\") "
Jul 29 12:03:37.350798 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.350776    2569 scope.go:117] "RemoveContainer" containerID="09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f"
Jul 29 12:03:37.351115 ip-10-0-129-80 kubenswrapper[2569]: E0729 12:03:37.351090    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f\": container with ID starting with 09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f not found: ID does not exist" containerID="09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f"
Jul 29 12:03:37.351198 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.351124    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f"} err="failed to get container status \"09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f\": rpc error: code = NotFound desc = could not find container \"09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f\": container with ID starting with 09d8ef380bf569040973bde69582c251f345a54bb8c70d808e2630ebb9f40b5f not found: ID does not exist"
Jul 29 12:03:37.352840 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.352810    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ab07f2-b993-4921-9054-a2148898824f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9ab07f2-b993-4921-9054-a2148898824f" (UID: "a9ab07f2-b993-4921-9054-a2148898824f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 12:03:37.352947 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.352881    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ab07f2-b993-4921-9054-a2148898824f-kube-api-access-j5q9r" (OuterVolumeSpecName: "kube-api-access-j5q9r") pod "a9ab07f2-b993-4921-9054-a2148898824f" (UID: "a9ab07f2-b993-4921-9054-a2148898824f"). InnerVolumeSpecName "kube-api-access-j5q9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:03:37.452154 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.452120    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab07f2-b993-4921-9054-a2148898824f-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 12:03:37.452154 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.452147    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5q9r\" (UniqueName: \"kubernetes.io/projected/a9ab07f2-b993-4921-9054-a2148898824f-kube-api-access-j5q9r\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 12:03:37.664283 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.664250    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"]
Jul 29 12:03:37.667527 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:37.667501    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-rrx5n"]
Jul 29 12:03:38.041917 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:03:38.041833    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ab07f2-b993-4921-9054-a2148898824f" path="/var/lib/kubelet/pods/a9ab07f2-b993-4921-9054-a2148898824f/volumes"
Jul 29 12:04:27.483059 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:27.483017    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:04:27.580719 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:27.580686    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:04:33.342787 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:33.342746    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:04:36.556032 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.555994    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:04:36.727972 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.727936    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"]
Jul 29 12:04:36.728626 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.728600    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ab07f2-b993-4921-9054-a2148898824f" containerName="main"
Jul 29 12:04:36.728626 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.728625    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ab07f2-b993-4921-9054-a2148898824f" containerName="main"
Jul 29 12:04:36.728766 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.728687    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ab07f2-b993-4921-9054-a2148898824f" containerName="main"
Jul 29 12:04:36.731620 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.731602    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:04:36.735447 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.735416    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-sbbnq\""
Jul 29 12:04:36.735565 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.735447    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 29 12:04:36.735565 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.735514    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 29 12:04:36.735565 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.735547    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 29 12:04:36.735868 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.735850    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-ntkdx\""
Jul 29 12:04:36.738523 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.738502    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"]
Jul 29 12:04:36.751784 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.751755    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:04:36.794757 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.794698    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-kube-api-access-92xdl\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr\" (UID: \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:04:36.795094 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.794801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr\" (UID: \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:04:36.896063 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.896021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-kube-api-access-92xdl\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr\" (UID: \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:04:36.896063 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.896067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr\" (UID: \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:04:36.898979 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.898940    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr\" (UID: \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:04:36.903993 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:36.903966    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-kube-api-access-92xdl\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr\" (UID: \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:04:37.042995 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:37.042964    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:04:37.181247 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:37.181164    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"]
Jul 29 12:04:37.184574 ip-10-0-129-80 kubenswrapper[2569]: W0729 12:04:37.184544    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16901f8e_8885_4fb2_b9e9_ee348f6c2a5f.slice/crio-fa298d5d579dfe6a0eff67fa03686f084d24d626ab55f29d97518f2c17b9794f WatchSource:0}: Error finding container fa298d5d579dfe6a0eff67fa03686f084d24d626ab55f29d97518f2c17b9794f: Status 404 returned error can't find the container with id fa298d5d579dfe6a0eff67fa03686f084d24d626ab55f29d97518f2c17b9794f
Jul 29 12:04:37.628979 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:37.628932    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr" event={"ID":"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f","Type":"ContainerStarted","Data":"d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021"}
Jul 29 12:04:37.629375 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:37.629000    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr" event={"ID":"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f","Type":"ContainerStarted","Data":"fa298d5d579dfe6a0eff67fa03686f084d24d626ab55f29d97518f2c17b9794f"}
Jul 29 12:04:37.629375 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:37.629116    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:04:37.645145 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:37.645092    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr" podStartSLOduration=1.6450733309999999 podStartE2EDuration="1.645073331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:04:37.643383153 +0000 UTC m=+5752.036596023" watchObservedRunningTime="2026-07-29 12:04:37.645073331 +0000 UTC m=+5752.038286201"
Jul 29 12:04:38.813921 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:38.813887    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:04:38.869733 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:04:38.869697    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:05:08.635922 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:05:08.635889    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:05:16.698419 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:05:16.698373    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:05:36.322611 ip-10-0-129-80 kubenswrapper[2569]: E0729 12:05:36.322560    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 12:05:36.916000 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:05:36.915962    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 12:06:07.469608 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:07.469571    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"]
Jul 29 12:06:07.471899 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:07.471865    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:06:08.635825 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:08.635731    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr" podUID="16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 12:06:15.327604 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:15.327565    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:06:18.635420 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:18.635376    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr" podUID="16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 12:06:22.489831 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:22.489775    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr" podUID="16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" containerName="main" containerID="cri-o://d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021" gracePeriod=45
Jul 29 12:06:22.755793 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:22.755726    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:06:22.819976 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:22.819935    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-tls-certs\") pod \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\" (UID: \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\") "
Jul 29 12:06:22.820173 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:22.820035    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-kube-api-access-92xdl\") pod \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\" (UID: \"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f\") "
Jul 29 12:06:22.822313 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:22.822284    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" (UID: "16901f8e-8885-4fb2-b9e9-ee348f6c2a5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 12:06:22.822313 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:22.822302    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-kube-api-access-92xdl" (OuterVolumeSpecName: "kube-api-access-92xdl") pod "16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" (UID: "16901f8e-8885-4fb2-b9e9-ee348f6c2a5f"). InnerVolumeSpecName "kube-api-access-92xdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:06:22.920972 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:22.920933    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-kube-api-access-92xdl\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 12:06:22.920972 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:22.920969    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\""
Jul 29 12:06:23.131230 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:23.131187    2569 generic.go:356] "Generic (PLEG): container finished" podID="16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" containerID="d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021" exitCode=0
Jul 29 12:06:23.131404 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:23.131253    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"
Jul 29 12:06:23.131404 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:23.131260    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr" event={"ID":"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f","Type":"ContainerDied","Data":"d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021"}
Jul 29 12:06:23.131404 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:23.131314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr" event={"ID":"16901f8e-8885-4fb2-b9e9-ee348f6c2a5f","Type":"ContainerDied","Data":"fa298d5d579dfe6a0eff67fa03686f084d24d626ab55f29d97518f2c17b9794f"}
Jul 29 12:06:23.131404 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:23.131344    2569 scope.go:117] "RemoveContainer" containerID="d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021"
Jul 29 12:06:23.140844 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:23.140823    2569 scope.go:117] "RemoveContainer" containerID="d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021"
Jul 29 12:06:23.141109 ip-10-0-129-80 kubenswrapper[2569]: E0729 12:06:23.141087    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021\": container with ID starting with d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021 not found: ID does not exist" containerID="d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021"
Jul 29 12:06:23.141163 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:23.141119    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021"} err="failed to get container status \"d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021\": rpc error: code = NotFound desc = could not find container \"d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021\": container with ID starting with d81432f82372257ade81a083e5cb4a937fb566b9a0e31873c92b757da6533021 not found: ID does not exist"
Jul 29 12:06:23.152849 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:23.152820    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"]
Jul 29 12:06:23.155585 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:23.155559    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-qwjvr"]
Jul 29 12:06:24.042645 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:24.042613    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" path="/var/lib/kubelet/pods/16901f8e-8885-4fb2-b9e9-ee348f6c2a5f/volumes"
Jul 29 12:06:24.999839 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:24.999799    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:06:25.100933 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:25.100890    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:06:25.199829 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:25.199794    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:06:30.914056 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:30.914016    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:06:36.375788 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:36.375747    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:06:36.434829 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:06:36.434791    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:07:10.658111 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:07:10.658076    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:07:39.916942 ip-10-0-129-80 kubenswrapper[2569]: E0729 12:07:39.916833    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 12:07:40.510694 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:07:40.510658    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 12:07:44.719056 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:07:44.719019    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:07:50.726930 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:07:50.726892    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:07:50.824439 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:07:50.824396    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:08:25.391645 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:08:25.391599    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:08:57.975964 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:08:57.975927    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:08:58.029020 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:08:58.028986    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:09:43.511217 ip-10-0-129-80 kubenswrapper[2569]: E0729 12:09:43.511164    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 12:09:44.097790 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:09:44.097757    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 12:10:00.104247 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:10:00.104202    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:11:39.633582 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:11:39.633545    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:11:41.283884 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:11:41.283845    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:11:47.099261 ip-10-0-129-80 kubenswrapper[2569]: E0729 12:11:47.099216    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk" podUID="db0a398f-6814-400e-bb81-fcae5bc445af"
Jul 29 12:11:47.694127 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:11:47.694093    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bcc5lk"
Jul 29 12:11:58.261167 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:11:58.261127    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:11:58.322406 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:11:58.322369    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:12:21.885177 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:21.885084    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46d7p_3a1895cc-3ece-4c53-9be4-784aa44da7ca/discovery/0.log"
Jul 29 12:12:21.972553 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:21.972523    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d6d96c7dc-qc8qb_b299a586-25a6-414c-8717-57184234e96d/router/0.log"
Jul 29 12:12:23.079378 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:23.079342    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46d7p_3a1895cc-3ece-4c53-9be4-784aa44da7ca/discovery/0.log"
Jul 29 12:12:23.170310 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:23.170276    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d6d96c7dc-qc8qb_b299a586-25a6-414c-8717-57184234e96d/router/0.log"
Jul 29 12:12:24.270231 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:24.270194    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46d7p_3a1895cc-3ece-4c53-9be4-784aa44da7ca/discovery/0.log"
Jul 29 12:12:24.355922 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:24.355890    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d6d96c7dc-qc8qb_b299a586-25a6-414c-8717-57184234e96d/router/0.log"
Jul 29 12:12:25.479034 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:25.479003    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46d7p_3a1895cc-3ece-4c53-9be4-784aa44da7ca/discovery/0.log"
Jul 29 12:12:25.606826 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:25.606792    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d6d96c7dc-qc8qb_b299a586-25a6-414c-8717-57184234e96d/router/0.log"
Jul 29 12:12:26.441073 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:26.441036    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:26.453595 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:26.453534    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:26.808613 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:26.808526    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:27.839531 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:27.839491    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:27.872273 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:27.872242    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:28.219248 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:28.219159    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:29.241054 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:29.241011    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:29.253304 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:29.253252    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:29.566908 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:29.566808    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:30.464854 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:30.464821    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:30.478219 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:30.478192    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:30.794772 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:30.794668    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:31.682168 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:31.682137    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:31.694142 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:31.694112    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:32.012342 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:32.012247    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:32.895444 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:32.895408    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:32.907987 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:32.907952    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:33.229380 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:33.229292    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:34.134569 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:34.134520    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:34.149313 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:34.149275    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:34.473290 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:34.473190    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:35.351415 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:35.351378    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:35.364001 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:35.363969    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:35.677715 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:35.677625    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:36.570740 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:36.570712    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:36.582608 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:36.582569    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:36.897499 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:36.897447    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:37.800198 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:37.800163    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:37.813780 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:37.813747    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:38.129580 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:38.129546    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:39.019268 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:39.019238    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:39.031188 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:39.031160    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:39.355051 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:39.355014    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:40.279874 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:40.279840    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:40.291922 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:40.291882    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:40.607915 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:40.607878    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:41.480833 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:41.480800    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:41.493119 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:41.493089    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:41.805025 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:41.804936    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:42.681802 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:42.681764    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:42.693499 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:42.693466    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:43.011187 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:43.011085    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:43.903476 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:43.903417    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:43.916747 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:43.916721    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:44.239219 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:44.239129    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:45.129762 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:45.129729    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:45.142655 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:45.142607    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:45.476774 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:45.476669    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:46.381237 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:46.381196    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:46.393580 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:46.393532    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:46.709218 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:46.709116    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:47.613102 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:47.613065    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:47.625632 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:47.625605    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:47.956763 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:47.956668    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:48.852989 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:48.852954    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:48.866250 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:48.866216    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:49.182341 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:49.182259    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:50.092748 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:50.092704    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:50.104491 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:50.104463    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:50.422868 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:50.422783    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:51.328555 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:51.328515    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:51.341172 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:51.341135    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:51.659630 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:51.659588    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:52.554575 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:52.554534    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:52.566536 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:52.566500    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:52.890703 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:52.890667    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:53.780744 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:53.780705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:53.795927 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:53.795893    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:54.109254 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:54.109220    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:54.978877 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:54.978847    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:54.993708 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:54.993680    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:55.315315 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:55.315216    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:56.209236 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:56.209202    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:12:56.222225 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:56.222189    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:12:56.537127 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:56.537035    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:12:58.497527 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:58.497483    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:12:58.680138 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:58.680093    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:12:58.777774 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:12:58.777681    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:13:01.620684 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:01.620633    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r778m_2285452e-2515-4e83-8aa2-6334577e0140/global-pull-secret-syncer/0.log"
Jul 29 12:13:01.697805 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:01.697769    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log"
Jul 29 12:13:01.842572 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:01.842536    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-80.ec2.internal_f186ee96a5b10264dea34206a4faa43e/haproxy/0.log"
Jul 29 12:13:04.267915 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:04.267872    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:13:04.368366 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:04.368310    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2kzz5_c50ec00e-12f4-41e9-be76-5ab3c1571fad/konnectivity-agent/0.log" limit=1024
Jul 29 12:13:05.746836 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:05.746805    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-rzd9v_3f55bcf0-6e72-49da-b427-04f7a3419901/manager/0.log"
Jul 29 12:13:05.775358 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:05.775322    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-9wzdw_1dedce6f-c591-4285-b6a4-b88862969921/manager/0.log"
Jul 29 12:13:06.126722 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:06.126692    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-m5dpg_0ed5e62a-0495-410f-bea8-c776f9bdee17/manager/0.log"
Jul 29 12:13:08.036906 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.036867    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-vb4q2_ff10a068-40c3-427e-bbcc-bc2012d9e43c/cluster-monitoring-operator/0.log"
Jul 29 12:13:08.067367 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.067341    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6ksth_0bd6173a-0f30-42cd-bf68-1dc9a65782ed/kube-state-metrics/0.log"
Jul 29 12:13:08.088065 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.088033    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6ksth_0bd6173a-0f30-42cd-bf68-1dc9a65782ed/kube-rbac-proxy-main/0.log"
Jul 29 12:13:08.111352 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.111329    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-6ksth_0bd6173a-0f30-42cd-bf68-1dc9a65782ed/kube-rbac-proxy-self/0.log"
Jul 29 12:13:08.144087 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.144055    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fb54f9c7d-2hmrd_f553caa1-3de3-4749-aba5-eed1765f3819/metrics-server/0.log"
Jul 29 12:13:08.365475 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.365442    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tpnck_e6c39967-9e25-46e3-88dd-d6f07e0559d8/node-exporter/0.log"
Jul 29 12:13:08.385585 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.385557    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tpnck_e6c39967-9e25-46e3-88dd-d6f07e0559d8/kube-rbac-proxy/0.log"
Jul 29 12:13:08.406772 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.406752    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tpnck_e6c39967-9e25-46e3-88dd-d6f07e0559d8/init-textfile/0.log"
Jul 29 12:13:08.884025 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.883996    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bbf8fcf7d-wk55s_9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb/thanos-query/0.log"
Jul 29 12:13:08.907854 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.907827    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bbf8fcf7d-wk55s_9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb/kube-rbac-proxy-web/0.log"
Jul 29 12:13:08.932672 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.932644    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bbf8fcf7d-wk55s_9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb/kube-rbac-proxy/0.log"
Jul 29 12:13:08.980277 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:08.980251    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bbf8fcf7d-wk55s_9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb/prom-label-proxy/0.log"
Jul 29 12:13:09.008949 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:09.008906    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bbf8fcf7d-wk55s_9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb/kube-rbac-proxy-rules/0.log"
Jul 29 12:13:09.040329 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:09.040302    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bbf8fcf7d-wk55s_9fd4be9e-2d45-4e19-a6ef-f8e94389b6cb/kube-rbac-proxy-metrics/0.log"
Jul 29 12:13:12.274019 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.273977    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"]
Jul 29 12:13:12.274566 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.274551    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" containerName="main"
Jul 29 12:13:12.274642 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.274573    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" containerName="main"
Jul 29 12:13:12.274685 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.274673    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="16901f8e-8885-4fb2-b9e9-ee348f6c2a5f" containerName="main"
Jul 29 12:13:12.279319 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.279275    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.282781 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.282734    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjmjm\"/\"openshift-service-ca.crt\""
Jul 29 12:13:12.282926 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.282882    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jjmjm\"/\"default-dockercfg-zcqgd\""
Jul 29 12:13:12.283646 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.283627    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjmjm\"/\"kube-root-ca.crt\""
Jul 29 12:13:12.286378 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.286351    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"]
Jul 29 12:13:12.316364 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.316335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-sys\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.316512 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.316466    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-proc\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.316512 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.316508    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-podres\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.316597 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.316528    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrffw\" (UniqueName: \"kubernetes.io/projected/5c2a8412-68ca-4242-9c61-00cb8d4cca18-kube-api-access-xrffw\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.316597 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.316584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-lib-modules\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.416978 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.416938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrffw\" (UniqueName: \"kubernetes.io/projected/5c2a8412-68ca-4242-9c61-00cb8d4cca18-kube-api-access-xrffw\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.417186 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.417001    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-lib-modules\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.417186 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.417057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-sys\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.417186 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.417139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-proc\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.417186 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.417168    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-sys\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.417186 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.417180    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-podres\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.417415 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.417198    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-lib-modules\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.417415 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.417255    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-proc\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.417415 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.417291    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c2a8412-68ca-4242-9c61-00cb8d4cca18-podres\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.424757 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.424731    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrffw\" (UniqueName: \"kubernetes.io/projected/5c2a8412-68ca-4242-9c61-00cb8d4cca18-kube-api-access-xrffw\") pod \"perf-node-gather-daemonset-4r6k6\" (UID: \"5c2a8412-68ca-4242-9c61-00cb8d4cca18\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.591037 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.591001    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:12.726965 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:12.726938    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"]
Jul 29 12:13:12.729263 ip-10-0-129-80 kubenswrapper[2569]: W0729 12:13:12.729229    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c2a8412_68ca_4242_9c61_00cb8d4cca18.slice/crio-a2ef8f86f6d3332b850d6cc5ea70bf2ff04e0b0d2e7060004219e209d50269f4 WatchSource:0}: Error finding container a2ef8f86f6d3332b850d6cc5ea70bf2ff04e0b0d2e7060004219e209d50269f4: Status 404 returned error can't find the container with id a2ef8f86f6d3332b850d6cc5ea70bf2ff04e0b0d2e7060004219e209d50269f4
Jul 29 12:13:13.104559 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:13.104526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6" event={"ID":"5c2a8412-68ca-4242-9c61-00cb8d4cca18","Type":"ContainerStarted","Data":"2f10741b067b0bad8647c7b35e268fb45f93f6157d4907ce5d7eac01cfd9c151"}
Jul 29 12:13:13.104773 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:13.104587    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6" event={"ID":"5c2a8412-68ca-4242-9c61-00cb8d4cca18","Type":"ContainerStarted","Data":"a2ef8f86f6d3332b850d6cc5ea70bf2ff04e0b0d2e7060004219e209d50269f4"}
Jul 29 12:13:13.104773 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:13.104635    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:13.122597 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:13.122542    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6" podStartSLOduration=1.122526123 podStartE2EDuration="1.122526123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 12:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:13:13.120537695 +0000 UTC m=+6267.513750565" watchObservedRunningTime="2026-07-29 12:13:13.122526123 +0000 UTC m=+6267.515738994"
Jul 29 12:13:13.209760 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:13.209731    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j5t5h_165e8d7d-5361-458b-91a6-8bea4c4b419a/dns/0.log"
Jul 29 12:13:13.231799 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:13.231770    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j5t5h_165e8d7d-5361-458b-91a6-8bea4c4b419a/kube-rbac-proxy/0.log"
Jul 29 12:13:13.346735 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:13.346704    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-57dwz_001cfcc8-dc86-4070-b7e7-f374fbae8581/dns-node-resolver/0.log"
Jul 29 12:13:13.948770 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:13.948737    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86496c8854-sxqfg_71bfecff-e8ba-46b8-9d39-a5995d536ce7/registry/0.log"
Jul 29 12:13:14.004458 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:14.004417    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8gj9q_a9a64c38-a6c8-4d27-9d16-77605f9f5742/node-ca/0.log"
Jul 29 12:13:15.362690 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:15.362649    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46d7p_3a1895cc-3ece-4c53-9be4-784aa44da7ca/discovery/0.log"
Jul 29 12:13:15.465478 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:15.465438    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d6d96c7dc-qc8qb_b299a586-25a6-414c-8717-57184234e96d/router/0.log"
Jul 29 12:13:15.967736 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:15.967705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-796bm_8701dd0c-e3d3-4f65-986c-04f2ee51bc34/serve-healthcheck-canary/0.log"
Jul 29 12:13:16.449910 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:16.449872    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-5dgxd_c2c7d066-1ffc-460b-88c0-709d3f265242/insights-operator/0.log"
Jul 29 12:13:16.452759 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:16.452731    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-5dgxd_c2c7d066-1ffc-460b-88c0-709d3f265242/insights-operator/1.log"
Jul 29 12:13:16.475475 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:16.475422    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2df6f_508e5425-0b4c-4b63-b670-dfb59eade57d/kube-rbac-proxy/0.log"
Jul 29 12:13:16.496952 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:16.496924    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2df6f_508e5425-0b4c-4b63-b670-dfb59eade57d/exporter/0.log"
Jul 29 12:13:16.519244 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:16.519210    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2df6f_508e5425-0b4c-4b63-b670-dfb59eade57d/extractor/0.log"
Jul 29 12:13:19.120086 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:19.120061    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-4r6k6"
Jul 29 12:13:19.568610 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:19.568469    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-rd9zg_5ad3fa79-f971-4c2f-9aee-4e7199326813/openshift-lws-operator/0.log"
Jul 29 12:13:20.146365 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:20.146336    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-65f566c74b-m6f8j_6f515603-6f8c-47f7-a49f-9f0ed77eaee1/manager/0.log"
Jul 29 12:13:20.236470 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:20.236418    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-bj279_918aa4b5-19e1-4fee-89a8-92fef107680e/server/0.log"
Jul 29 12:13:21.513144 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:21.513105    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-6h4sv_3f50e258-57c9-4278-a06d-25d8c49a8719/manager/0.log"
Jul 29 12:13:21.648993 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:21.648903    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-r6dnp_170bf724-f8cc-45e8-9649-bf48faeef1ba/seaweedfs/0.log"
Jul 29 12:13:21.706751 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:21.706714    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46d7p_3a1895cc-3ece-4c53-9be4-784aa44da7ca/discovery/0.log"
Jul 29 12:13:21.811834 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:21.811804    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d6d96c7dc-qc8qb_b299a586-25a6-414c-8717-57184234e96d/router/0.log"
Jul 29 12:13:23.361507 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:23.361467    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46d7p_3a1895cc-3ece-4c53-9be4-784aa44da7ca/discovery/0.log"
Jul 29 12:13:23.454073 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:23.454041    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d6d96c7dc-qc8qb_b299a586-25a6-414c-8717-57184234e96d/router/0.log"
Jul 29 12:13:28.048932 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.048899    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6nndf_262e51a4-76b2-4a18-9814-9dd42ccf84fd/kube-multus/0.log"
Jul 29 12:13:28.147438 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.147405    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-45d2k_32248a0a-b444-475d-8a21-2139f1dde3fa/kube-multus-additional-cni-plugins/0.log"
Jul 29 12:13:28.170004 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.169975    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-45d2k_32248a0a-b444-475d-8a21-2139f1dde3fa/egress-router-binary-copy/0.log"
Jul 29 12:13:28.192235 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.192208    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-45d2k_32248a0a-b444-475d-8a21-2139f1dde3fa/cni-plugins/0.log"
Jul 29 12:13:28.216346 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.216310    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-45d2k_32248a0a-b444-475d-8a21-2139f1dde3fa/bond-cni-plugin/0.log"
Jul 29 12:13:28.238612 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.238582    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-45d2k_32248a0a-b444-475d-8a21-2139f1dde3fa/routeoverride-cni/0.log"
Jul 29 12:13:28.259091 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.259063    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-45d2k_32248a0a-b444-475d-8a21-2139f1dde3fa/whereabouts-cni-bincopy/0.log"
Jul 29 12:13:28.279773 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.279746    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-45d2k_32248a0a-b444-475d-8a21-2139f1dde3fa/whereabouts-cni/0.log"
Jul 29 12:13:28.750964 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.750931    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ddfml_d14e709b-eb10-4251-80dc-54e1aa91e1dd/network-metrics-daemon/0.log"
Jul 29 12:13:28.769320 ip-10-0-129-80 kubenswrapper[2569]: I0729 12:13:28.769286    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ddfml_d14e709b-eb10-4251-80dc-54e1aa91e1dd/kube-rbac-proxy/0.log"