Jul 28 17:21:00.093130 ip-10-0-133-142 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:21:00.093146 ip-10-0-133-142 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:21:00.093156 ip-10-0-133-142 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:21:00.093475 ip-10-0-133-142 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:21:09.793224 ip-10-0-133-142 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 28 17:21:09.793238 ip-10-0-133-142 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 71d17291e8a84d859d6ba985ca12d7f2 --
Jul 28 17:22:38.153167 ip-10-0-133-142 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:22:38.598058 ip-10-0-133-142 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:22:38.598058 ip-10-0-133-142 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:22:38.598058 ip-10-0-133-142 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:22:38.598058 ip-10-0-133-142 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:22:38.598058 ip-10-0-133-142 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:22:38.600272 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.600117    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:22:38.604319 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604302    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:38.604319 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604320    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604324    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604327    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604331    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604334    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604337    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604340    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604343    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604346    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604349    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604352    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604356    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604360    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604363    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604367    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604370    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604372    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604375    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604377    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:38.604387 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604380    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604383    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604385    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604388    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604390    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604393    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604396    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604399    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604402    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604405    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604407    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604410    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604412    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604414    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604417    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604419    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604422    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604424    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604427    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604430    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:38.604827 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604433    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604436    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604440    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604442    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604445    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604447    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604450    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604452    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604454    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604457    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604459    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604462    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604466    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604469    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604473    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604476    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604479    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604482    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604485    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:38.605322 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604488    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604490    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604492    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604495    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604497    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604500    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604502    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604505    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604507    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604510    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604512    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604514    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604518    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604520    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604522    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604525    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604527    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604530    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604532    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604534    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:38.605794 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604537    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604539    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604542    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604544    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604546    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604549    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604552    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604555    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604558    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604560    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604563    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604566    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604568    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604571    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604573    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604576    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604578    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604581    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604583    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.604585    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:38.606283 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605288    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605295    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605298    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605300    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605303    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605306    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605309    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605311    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605314    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605317    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605319    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605322    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605325    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605327    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605331    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605333    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605336    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605339    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605343    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:38.606766 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605349    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605352    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605354    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605357    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605360    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605363    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605365    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605367    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605370    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605373    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605375    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605377    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605380    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605382    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605385    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605388    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605390    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605392    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605395    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605398    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:38.607247 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605401    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605403    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605406    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605408    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605411    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605413    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605415    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605418    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605420    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605423    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605425    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605428    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605431    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605434    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605436    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605439    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605447    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605450    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605453    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605455    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:38.607731 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605458    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605460    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605463    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605465    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605469    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605472    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605475    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605478    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605481    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605484    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605487    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605489    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605493    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605496    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605508    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605512    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605514    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605517    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605519    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:38.608236 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605523    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605525    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605528    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605530    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605533    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605539    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605542    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605545    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605547    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605550    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605552    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605555    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605558    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605560    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605563    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605565    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605568    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605571    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605573    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605576    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:38.608683 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605579    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605977    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605983    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605986    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605988    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605992    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605995    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.605998    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606000    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606003    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606005    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606008    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606010    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606013    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606016    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606018    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606021    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606024    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606027    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606029    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606032    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:38.609184 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606035    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606037    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606040    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606042    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606045    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606047    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606050    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606052    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606055    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606057    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606060    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606062    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606065    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606068    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606070    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606073    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606075    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606078    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606081    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:38.609817 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606083    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606086    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606088    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606090    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606093    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606096    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606098    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606100    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606104    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606107    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606109    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606112    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606115    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606117    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606119    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606122    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606124    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606127    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606129    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606131    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:38.610280 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606134    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606136    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606139    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606142    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606145    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606148    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606151    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606153    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606156    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606159    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606162    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606165    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606167    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606170    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606173    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606175    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606180    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606183    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606185    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:38.610819 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606188    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606190    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606193    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606195    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606197    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606200    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606202    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606205    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606207    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606210    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606212    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606215    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606217    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606220    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606224    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606227    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606230    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606232    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606235    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:38.611313 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606237    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606241    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606313    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606320    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606331    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606335    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606340    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606343    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606347    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606352    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606356    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606359    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606363    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606366    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606369    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606372    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606375    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606377    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606380    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606383    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606387    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606390    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606393    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:22:38.611805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606396    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606400    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606404    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606407    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606410    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606413    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606416    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606419    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606422    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606425    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606428    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606432    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606435    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606438    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606440    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606443    2559 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606447    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606452    2559 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606455    2559 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606459    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606462    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606465    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606469    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606472    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606475    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:22:38.612366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606478    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606480    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606483    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606486    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606489    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606492    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606495    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606498    2559 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606502    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606505    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606508    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606511    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606514    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606517    2559 flags.go:64] FLAG: --help="false"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606520    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606523    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606526    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606529    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606532    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606535    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606538    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606541    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606544    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:22:38.612961 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606547    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606556    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606559    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606563    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606566    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606569    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606572    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606575    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606578    2559 flags.go:64] FLAG: --lock-file=""
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606580    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606583    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606586    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606591    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606594    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606597    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606599    2559 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606603    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606606    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606609    2559 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606611    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606615    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606618    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606622    2559 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606625    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606627    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:22:38.613543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606630    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606633    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606636    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606638    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606641    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606649    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606652    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606655    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606658    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606666    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606672    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606676    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606679    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606681    2559 flags.go:64] FLAG: --port="10250"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606684    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606687    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0270663637aff0771"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606690    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606693    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606696    2559 flags.go:64] FLAG: --register-node="true"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606699    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606703    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606706    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606708    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606711    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606714    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:22:38.614182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606717    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606720    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606723    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606726    2559 flags.go:64] FLAG: --runonce="false"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606728    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606731    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606734    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606738    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606741    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606743    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606746    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606751    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606754    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606757    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606759    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606762    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606765    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606773    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606777    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606783    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606786    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606789    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606793    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606796    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606798    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:22:38.614793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606801    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606804    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606806    2559 flags.go:64] FLAG: --v="2"
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606810    2559 flags.go:64] FLAG: --version="false"
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606814    2559 flags.go:64] FLAG: --vmodule=""
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606818    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.606821    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606978    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606983    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606986    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606990    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606994    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606997    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.606999    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607002    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607005    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607007    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607010    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607015    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607017    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607020    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:38.615447 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607023    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607026    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607029    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607031    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607045    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607048    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607050    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607053    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607055    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607058    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607061    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607063    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607065    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607068    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607071    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607073    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607076    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607078    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607080    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607083    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:38.615967 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607086    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607088    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607091    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607093    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607096    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607098    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607101    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607103    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607105    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607110    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607112    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607115    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607117    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607121    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607124    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607127    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607131    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607139    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607142    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:38.616438 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607145    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607147    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607150    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607152    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607155    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607157    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607160    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607162    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607165    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607167    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607170    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607172    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607175    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607177    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607180    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607183    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607185    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607189    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607191    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607194    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:38.616930 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607196    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607199    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607203    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607206    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607208    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607210    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607213    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607215    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607217    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607221    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607223    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607231    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607234    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607236    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607239    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607241    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607244    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607246    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607249    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607252    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:38.617416 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607254    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607257    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607259    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607262    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607265    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.607267    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.608051    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.615683    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.615698    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.616555    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.616567    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616626    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616631    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616634    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:38.617907 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616637    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616641    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616645    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616648    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616651    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616653    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616656    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616658    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616661    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616664    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616666    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616669    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616671    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616674    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616676    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616679    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616682    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616684    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616687    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616689    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:38.618264 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616692    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616694    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616697    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616699    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616703    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616706    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616712    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616715    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616718    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616721    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616723    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616726    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616728    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616731    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616734    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616736    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616739    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616741    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616744    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616746    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:38.618751 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616748    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616751    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616753    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616756    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616759    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616761    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616763    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616766    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616768    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616772    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616775    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616778    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616780    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616783    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616785    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616788    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616790    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616793    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616796    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616800    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:38.619245 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616802    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616805    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616807    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616810    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616812    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616814    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616817    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616820    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616822    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616824    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616827    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616830    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616832    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616835    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616838    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616840    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616842    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616845    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616848    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616850    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:38.619725 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616869    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616874    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616876    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616879    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616882    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616884    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616886    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616889    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616892    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616894    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616898    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616900    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616903    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616906    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616909    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:38.620287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.616911    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.616916    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617039    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617044    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617047    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617050    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617053    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617055    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617058    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617061    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617063    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617065    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617068    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617070    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:22:38.620649 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617073    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617075    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617077    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617080    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617082    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617084    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617087    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617089    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617092    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617094    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617097    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617099    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617101    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617104    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617107    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617110    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617113    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617116    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617118    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617120    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:22:38.621054 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617123    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617125    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617128    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617130    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617133    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617135    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617137    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617140    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617142    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617145    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617147    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617150    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617152    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617154    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617156    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617159    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617161    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617164    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617166    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617168    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:22:38.621560 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617171    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617173    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617176    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617178    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617180    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617183    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617185    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617188    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617190    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617193    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617196    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617198    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617200    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617203    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617205    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617207    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617210    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617212    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617214    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617217    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:22:38.622068 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617219    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617222    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617225    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617228    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617232    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617234    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617237    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617240    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617242    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617245    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617248    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617250    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617253    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617256    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617258    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617261    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617264    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617268    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617272    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:22:38.622563 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617275    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617278    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617282    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617286    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617288    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617290    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617293    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:38.617295    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.617300    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.617408    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.619814    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.621570    2559 server.go:1019] "Starting client certificate rotation"
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.621670    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:22:38.623032 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.621708    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:22:38.624337 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.624323    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:22:38.628468 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.628448    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:22:38.636224 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.636206    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:22:38.640294 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.640278    2559 log.go:25] "Validated CRI v1 image API"
Jul 28 17:22:38.642893 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.642874    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:22:38.645591 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.645574    2559 fs.go:135] Filesystem UUIDs: map[4a0d52b7-9245-4d11-9278-cd919efbf1d6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 db2433b1-a32a-4c7b-8588-62119c2a33f7:/dev/nvme0n1p4]
Jul 28 17:22:38.645629 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.645592    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:22:38.651016 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.650848    2559 manager.go:216] Machine: {Timestamp:2026-07-28 17:22:38.649360935 +0000 UTC m=+0.366873680 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096045 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22d0d9420bd0b3dd069bac4ba0e639 SystemUUID:ec22d0d9-420b-d0b3-dd06-9bac4ba0e639 BootID:71d17291-e8a8-4d85-9d6b-a985ca12d7f2 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a5:3c:03:0b:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a5:3c:03:0b:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:36:01:e1:a9:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:22:38.651016 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.651012    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:22:38.651126 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.651049    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:22:38.651126 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.651119    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:22:38.653103 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.653078    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:22:38.653235 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.653106    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:22:38.654706 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.654694    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:22:38.654706 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.654707    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:22:38.654787 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.654714    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:22:38.655618 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.655608    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:22:38.657535 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.657525    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:22:38.662033 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.662022    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:22:38.662149 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.662140    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:22:38.662185 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.662171    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:22:38.662185 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.662180    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:22:38.662237 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.662194    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:22:38.662237 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.662204    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:22:38.662237 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.662213    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:22:38.662694 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.662682    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:22:38.662752 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.662710    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:22:38.665232 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.665205    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:22:38.667247 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.667201    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:22:38.667291 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.667261    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:22:38.668151 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668140    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:22:38.668181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668156    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:22:38.668181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668163    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:22:38.668181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668168    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:22:38.668181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668175    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:22:38.668181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668181    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:22:38.668312 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668198    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:22:38.668312 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668203    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:22:38.668312 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668217    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:22:38.668312 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668223    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:22:38.668312 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668231    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:22:38.668312 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.668238    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:22:38.669101 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.669092    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:22:38.669101 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.669101    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:22:38.673824 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.673805    2559 server.go:1268] "Started kubelet"
Jul 28 17:22:38.673990 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.673969    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:22:38.674059 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.673961    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:22:38.674059 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.674030    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:22:38.674661 ip-10-0-133-142 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:22:38.676120 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.676101    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:22:38.676211 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.676130    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:22:38.677720 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.677694    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:22:38.677800 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.677782    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:22:38.677893 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.677847    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:22:38.678919 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.678902    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:22:38.682982 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.681817    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-142.ec2.internal.18c683ed91b10b51  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-142.ec2.internal,UID:ip-10-0-133-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-142.ec2.internal,},FirstTimestamp:2026-07-28 17:22:38.673767249 +0000 UTC m=+0.391279994,LastTimestamp:2026-07-28 17:22:38.673767249 +0000 UTC m=+0.391279994,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-142.ec2.internal,}"
Jul 28 17:22:38.686309 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.686296    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:22:38.686355 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.686335    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:22:38.686421 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.686404    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:38.687083 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687054    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:22:38.687181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687056    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:22:38.687181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687098    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:22:38.687281 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687214    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:22:38.687281 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687229    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:22:38.687440 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687406    2559 factory.go:55] Registering systemd factory
Jul 28 17:22:38.687440 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687431    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:22:38.687684 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687672    2559 factory.go:153] Registering CRI-O factory
Jul 28 17:22:38.687737 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687691    2559 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:22:38.687768 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687743    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:22:38.687768 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687759    2559 factory.go:103] Registering Raw factory
Jul 28 17:22:38.687833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.687774    2559 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:22:38.688834 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.688790    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:22:38.688933 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.688810    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:22:38.689326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.689312    2559 manager.go:349] Starting recovery of all containers
Jul 28 17:22:38.702078 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.702057    2559 manager.go:354] Recovery completed
Jul 28 17:22:38.706539 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.706522    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58fg4"
Jul 28 17:22:38.707365 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.707353    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:38.710020 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.710005    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:38.710077 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.710033    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:38.710077 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.710044    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:38.710604 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.710582    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:22:38.710604 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.710599    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:22:38.710738 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.710587    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:38.710738 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.710619    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:22:38.711754 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.711739    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:38.711804 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.711766    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:38.711804 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.711776    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:38.712738 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.712726    2559 policy_none.go:49] "None policy: Start"
Jul 28 17:22:38.712772 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.712745    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:22:38.712772 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.712756    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:22:38.714799 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.714784    2559 policy_none.go:47] "Start"
Jul 28 17:22:38.714898 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.714884    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58fg4"
Jul 28 17:22:38.721827 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.721663    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-142.ec2.internal.18c683ed93da39f9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-142.ec2.internal,UID:ip-10-0-133-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-142.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-142.ec2.internal,},FirstTimestamp:2026-07-28 17:22:38.710020601 +0000 UTC m=+0.427533349,LastTimestamp:2026-07-28 17:22:38.710020601 +0000 UTC m=+0.427533349,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-142.ec2.internal,}"
Jul 28 17:22:38.751431 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.751417    2559 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.751457    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.751470    2559 server.go:87] "Starting device plugin registration server"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.751732    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.751747    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.751841    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.751970    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.751979    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.752467    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.752524    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.762411    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.763660    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.763704    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.763728    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.763784    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:22:38.773292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.766488    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:38.852044 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.851971    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:38.853194 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.853178    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:38.853286 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.853206    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:38.853286 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.853229    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:38.853286 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.853257    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.863895 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.863875    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.863970 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.863895    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-142.ec2.internal\": node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:38.864017 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.863970    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal"]
Jul 28 17:22:38.864049 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.864031    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:38.865587 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.865567    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:38.865689 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.865599    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:38.865689 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.865609    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:38.867993 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.867977    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:38.868103 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.868089    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.868153 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.868121    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:38.868729 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.868714    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:38.868789 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.868748    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:38.868789 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.868761    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:38.868789 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.868788    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:38.868912 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.868797    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:38.868912 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.868764    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:38.871079 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.871064    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.871152 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.871095    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:38.872376 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.872361    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:38.872449 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.872386    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:38.872449 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.872410    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:38.878464 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.878449    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:38.888643 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.888630    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab14e603e28189365da9a7adafc8be56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal\" (UID: \"ab14e603e28189365da9a7adafc8be56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.888709 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.888651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab14e603e28189365da9a7adafc8be56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal\" (UID: \"ab14e603e28189365da9a7adafc8be56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.888709 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.888670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f12b46d1950955bc6884ff5ac3c8c404-config\") pod \"kube-apiserver-proxy-ip-10-0-133-142.ec2.internal\" (UID: \"f12b46d1950955bc6884ff5ac3c8c404\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.893487 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.893468    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-142.ec2.internal\" not found" node="ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.897824 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.897808    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-142.ec2.internal\" not found" node="ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.979584 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:38.979504    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:38.989075 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.989049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab14e603e28189365da9a7adafc8be56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal\" (UID: \"ab14e603e28189365da9a7adafc8be56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.989167 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.989082    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab14e603e28189365da9a7adafc8be56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal\" (UID: \"ab14e603e28189365da9a7adafc8be56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.989167 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.989105    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f12b46d1950955bc6884ff5ac3c8c404-config\") pod \"kube-apiserver-proxy-ip-10-0-133-142.ec2.internal\" (UID: \"f12b46d1950955bc6884ff5ac3c8c404\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.989167 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.989143    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab14e603e28189365da9a7adafc8be56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal\" (UID: \"ab14e603e28189365da9a7adafc8be56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.989279 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.989143    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab14e603e28189365da9a7adafc8be56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal\" (UID: \"ab14e603e28189365da9a7adafc8be56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:38.989279 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:38.989137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f12b46d1950955bc6884ff5ac3c8c404-config\") pod \"kube-apiserver-proxy-ip-10-0-133-142.ec2.internal\" (UID: \"f12b46d1950955bc6884ff5ac3c8c404\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:39.080331 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.080294    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:39.181091 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.181057    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:39.196522 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.196495    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:39.202812 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.202517    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:39.281637 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.281553    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:39.382136 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.382102    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:39.482619 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.482585    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:39.583130 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.583106    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:39.620578 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.620549    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:22:39.621003 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.620674    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:22:39.684080 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.684053    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:39.694337 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.694319    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:39.720518 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.720469    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:17:39 +0000 UTC" deadline="2028-04-04 20:57:33.300928928 +0000 UTC"
Jul 28 17:22:39.720518 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.720516    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14787h34m53.580416828s"
Jul 28 17:22:39.784570 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.784546    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:39.820077 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.820044    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:22:39.821288 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.821267    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:22:39.821444 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.821302    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:22:39.821444 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.821318    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:22:39.885609 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.885541    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:39.910705 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:39.910658    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab14e603e28189365da9a7adafc8be56.slice/crio-19721a9c908cacecec3f0b5442c4176047e85137bcc9a4b950489f08a96c0043 WatchSource:0}: Error finding container 19721a9c908cacecec3f0b5442c4176047e85137bcc9a4b950489f08a96c0043: Status 404 returned error can't find the container with id 19721a9c908cacecec3f0b5442c4176047e85137bcc9a4b950489f08a96c0043
Jul 28 17:22:39.910966 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:39.910943    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12b46d1950955bc6884ff5ac3c8c404.slice/crio-81dae52cffabfdc43ccde1a0bb06d97160725aeb888a27186a1e01b6ac499aca WatchSource:0}: Error finding container 81dae52cffabfdc43ccde1a0bb06d97160725aeb888a27186a1e01b6ac499aca: Status 404 returned error can't find the container with id 81dae52cffabfdc43ccde1a0bb06d97160725aeb888a27186a1e01b6ac499aca
Jul 28 17:22:39.915604 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:39.915589    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:22:39.985661 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:39.985621    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:40.086139 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:40.086109    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:40.186947 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:40.186841    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-142.ec2.internal\" not found"
Jul 28 17:22:41.288740 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.288720    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:41.295161 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.295136    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:41.350646 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.350606    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:41.372774 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.372745    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:22:41.373926 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.373910    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal"
Jul 28 17:22:41.392207 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.390931    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:22:41.657081 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.657054    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:22:41.726297 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.726265    2559 apiserver.go:52] "Watching apiserver"
Jul 28 17:22:41.735953 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.735926    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:22:41.736420 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.736398    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rff9b","kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal","openshift-cluster-node-tuning-operator/tuned-shs4g","openshift-dns/node-resolver-mtzfw","openshift-image-registry/node-ca-986jm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal","openshift-network-operator/iptables-alerter-8gs4b","openshift-ovn-kubernetes/ovnkube-node-nrwj7","kube-system/konnectivity-agent-2ffrd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl","openshift-multus/multus-additional-cni-plugins-dxvgt","openshift-multus/multus-zcfck","openshift-multus/network-metrics-daemon-89z9z","openshift-network-diagnostics/network-check-target-jzkpr"]
Jul 28 17:22:41.739345 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.739323    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.742186 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.742163    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.746960 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.743390    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pmflc\""
Jul 28 17:22:41.746960 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.743572    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:22:41.746960 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.743770    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:22:41.746960 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.743970    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.747570    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.747767    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.747975    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rk7lk\""
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.748163    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.748168    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.748277    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.748365    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.748633    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.748686    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:41.749428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.749294    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:22:41.752490 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.750744    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:22:41.752490 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.751046    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.752490 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.751856    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:22:41.752490 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.751966    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:22:41.752490 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.752102    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bmc4t\""
Jul 28 17:22:41.752490 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.752326    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:22:41.752826 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.752576    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:22:41.752826 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.752723    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:22:41.752826 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.752814    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zs6l6\""
Jul 28 17:22:41.753019 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.752993    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.753829 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.753256    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:22:41.753829 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.753488    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:22:41.754515 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.754085    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:22:41.754515 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.754127    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f424t\""
Jul 28 17:22:41.755633 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.754904    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:22:41.755633 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.755178    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:22:41.756365 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.756115    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:22:41.756778 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.756761    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:22:41.756979 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.756966    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n7647\""
Jul 28 17:22:41.757552 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.757144    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:22:41.757552 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.757311    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:22:41.757697 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.757604    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:22:41.759036 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.758502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.761677 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c6f3233-64d4-424d-9f24-b583d6beacf7-dbus\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:41.761677 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761656    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-etc-openvswitch\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.761815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.761815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761711    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnkb\" (UniqueName: \"kubernetes.io/projected/98966b68-fcf4-411b-9db9-20fd9012cd26-kube-api-access-tdnkb\") pod \"node-resolver-mtzfw\" (UID: \"98966b68-fcf4-411b-9db9-20fd9012cd26\") " pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.761815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68af73e0-884e-457e-9e5b-17a3ded1a5c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.761815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87bl\" (UniqueName: \"kubernetes.io/projected/6471e6ef-3d99-4bd7-9cb3-6abca723c71f-kube-api-access-c87bl\") pod \"iptables-alerter-8gs4b\" (UID: \"6471e6ef-3d99-4bd7-9cb3-6abca723c71f\") " pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.761815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-kubelet\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.761815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-run-systemd\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-cni-netd\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19f50166-5ce1-4963-b621-8366bf342528-env-overrides\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761869    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19f50166-5ce1-4963-b621-8366bf342528-ovnkube-script-lib\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk2vg\" (UniqueName: \"kubernetes.io/projected/19f50166-5ce1-4963-b621-8366bf342528-kube-api-access-nk2vg\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6j6\" (UniqueName: \"kubernetes.io/projected/68af73e0-884e-457e-9e5b-17a3ded1a5c3-kube-api-access-vq6j6\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761974    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-slash\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.761994    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-run-ovn-kubernetes\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-registration-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-sys-fs\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762087    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c6f3233-64d4-424d-9f24-b583d6beacf7-kubelet-config\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:41.762110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762107    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6471e6ef-3d99-4bd7-9cb3-6abca723c71f-iptables-alerter-script\") pod \"iptables-alerter-8gs4b\" (UID: \"6471e6ef-3d99-4bd7-9cb3-6abca723c71f\") " pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-log-socket\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19f50166-5ce1-4963-b621-8366bf342528-ovnkube-config\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-device-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/68af73e0-884e-457e-9e5b-17a3ded1a5c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-run-netns\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762263    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-run-openvswitch\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-run-ovn\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762319    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-cni-bin\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46kg\" (UniqueName: \"kubernetes.io/projected/163783e7-8154-4d7d-abc4-c4804e868dca-kube-api-access-f46kg\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-system-cni-dir\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4382b057-9b5d-43fb-8d3e-a64b33732a92-serviceca\") pod \"node-ca-986jm\" (UID: \"4382b057-9b5d-43fb-8d3e-a64b33732a92\") " pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6471e6ef-3d99-4bd7-9cb3-6abca723c71f-host-slash\") pod \"iptables-alerter-8gs4b\" (UID: \"6471e6ef-3d99-4bd7-9cb3-6abca723c71f\") " pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-var-lib-openvswitch\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19f50166-5ce1-4963-b621-8366bf342528-ovn-node-metrics-cert\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.762670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-socket-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762579    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-etc-selinux\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762597    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/98966b68-fcf4-411b-9db9-20fd9012cd26-hosts-file\") pod \"node-resolver-mtzfw\" (UID: \"98966b68-fcf4-411b-9db9-20fd9012cd26\") " pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762614    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-cnibin\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-systemd-units\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-node-log\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98966b68-fcf4-411b-9db9-20fd9012cd26-tmp-dir\") pod \"node-resolver-mtzfw\" (UID: \"98966b68-fcf4-411b-9db9-20fd9012cd26\") " pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-os-release\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762741    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/68af73e0-884e-457e-9e5b-17a3ded1a5c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4382b057-9b5d-43fb-8d3e-a64b33732a92-host\") pod \"node-ca-986jm\" (UID: \"4382b057-9b5d-43fb-8d3e-a64b33732a92\") " pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbpn\" (UniqueName: \"kubernetes.io/projected/4382b057-9b5d-43fb-8d3e-a64b33732a92-kube-api-access-pjbpn\") pod \"node-ca-986jm\" (UID: \"4382b057-9b5d-43fb-8d3e-a64b33732a92\") " pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.763348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.762909    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.764330 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.764311    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:41.764411 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.764392    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:41.764891 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.764857    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:22:41.765101 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.765080    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:22:41.765277 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.765253    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jtx85\""
Jul 28 17:22:41.768135 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.766975    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z9dgz\""
Jul 28 17:22:41.768135 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.767125    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:22:41.768135 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.767305    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:22:41.768135 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.767707    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:41.768135 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.767890    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gk5j7\""
Jul 28 17:22:41.768135 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.767956    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:22:41.768508 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.768240    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:22:41.770675 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.770654    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:22:41.788845 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.788826    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hb977"
Jul 28 17:22:41.802405 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.802382    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hb977"
Jul 28 17:22:41.833678 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.833629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal" event={"ID":"f12b46d1950955bc6884ff5ac3c8c404","Type":"ContainerStarted","Data":"81dae52cffabfdc43ccde1a0bb06d97160725aeb888a27186a1e01b6ac499aca"}
Jul 28 17:22:41.836010 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.835986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal" event={"ID":"ab14e603e28189365da9a7adafc8be56","Type":"ContainerStarted","Data":"19721a9c908cacecec3f0b5442c4176047e85137bcc9a4b950489f08a96c0043"}
Jul 28 17:22:41.851813 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.851793    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:22:41.863112 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863086    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhq2z\" (UniqueName: \"kubernetes.io/projected/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-kube-api-access-jhq2z\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.863208 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:41.863208 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-os-release\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.863208 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbpn\" (UniqueName: \"kubernetes.io/projected/4382b057-9b5d-43fb-8d3e-a64b33732a92-kube-api-access-pjbpn\") pod \"node-ca-986jm\" (UID: \"4382b057-9b5d-43fb-8d3e-a64b33732a92\") " pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.863208 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-run-k8s-cni-cncf-io\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.863392 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.863392 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnkb\" (UniqueName: \"kubernetes.io/projected/98966b68-fcf4-411b-9db9-20fd9012cd26-kube-api-access-tdnkb\") pod \"node-resolver-mtzfw\" (UID: \"98966b68-fcf4-411b-9db9-20fd9012cd26\") " pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.863392 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-cnibin\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.863392 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863302    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-os-release\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.863392 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-sysctl-conf\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.863392 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863346    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-host\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.863550 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68af73e0-884e-457e-9e5b-17a3ded1a5c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.863550 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87bl\" (UniqueName: \"kubernetes.io/projected/6471e6ef-3d99-4bd7-9cb3-6abca723c71f-kube-api-access-c87bl\") pod \"iptables-alerter-8gs4b\" (UID: \"6471e6ef-3d99-4bd7-9cb3-6abca723c71f\") " pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.863550 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-kubelet\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.863550 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-run-systemd\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.863682 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19f50166-5ce1-4963-b621-8366bf342528-env-overrides\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.863682 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk2vg\" (UniqueName: \"kubernetes.io/projected/19f50166-5ce1-4963-b621-8366bf342528-kube-api-access-nk2vg\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.863682 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-registration-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.863682 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46kg\" (UniqueName: \"kubernetes.io/projected/163783e7-8154-4d7d-abc4-c4804e868dca-kube-api-access-f46kg\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.863682 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-slash\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.863819 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-cni-binary-copy\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.863819 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-run-netns\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.863819 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863794    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-var-lib-kubelet\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.863819 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-hostroot\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.863986 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-modprobe-d\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.863986 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-lib-modules\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.863986 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.863986 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c6f3233-64d4-424d-9f24-b583d6beacf7-kubelet-config\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:41.863986 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863962    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6471e6ef-3d99-4bd7-9cb3-6abca723c71f-iptables-alerter-script\") pod \"iptables-alerter-8gs4b\" (UID: \"6471e6ef-3d99-4bd7-9cb3-6abca723c71f\") " pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.864147 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.863994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-log-socket\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.864147 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-device-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.864147 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-system-cni-dir\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.864147 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-etc-kubernetes\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.864147 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-sys\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.864147 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/68af73e0-884e-457e-9e5b-17a3ded1a5c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.864348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-run-netns\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.864348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864187    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-run-openvswitch\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.864348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864212    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.864348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-daemon-config\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.864348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-kubernetes\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.864348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-system-cni-dir\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.864348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4382b057-9b5d-43fb-8d3e-a64b33732a92-serviceca\") pod \"node-ca-986jm\" (UID: \"4382b057-9b5d-43fb-8d3e-a64b33732a92\") " pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.864611 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6471e6ef-3d99-4bd7-9cb3-6abca723c71f-host-slash\") pod \"iptables-alerter-8gs4b\" (UID: \"6471e6ef-3d99-4bd7-9cb3-6abca723c71f\") " pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.864611 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-var-lib-openvswitch\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.864611 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-socket-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.864611 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/98966b68-fcf4-411b-9db9-20fd9012cd26-hosts-file\") pod \"node-resolver-mtzfw\" (UID: \"98966b68-fcf4-411b-9db9-20fd9012cd26\") " pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.864611 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-conf-dir\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.864611 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-systemd\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.864611 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-systemd-units\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.864611 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-node-log\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.864611 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/845662cd-3d72-4f01-bf8f-2d448337f140-etc-tuned\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/68af73e0-884e-457e-9e5b-17a3ded1a5c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4382b057-9b5d-43fb-8d3e-a64b33732a92-host\") pod \"node-ca-986jm\" (UID: \"4382b057-9b5d-43fb-8d3e-a64b33732a92\") " pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864688    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-var-lib-cni-bin\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864710    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-run\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4153a55-9fee-476f-a80a-155d275f14e1-konnectivity-ca\") pod \"konnectivity-agent-2ffrd\" (UID: \"e4153a55-9fee-476f-a80a-155d275f14e1\") " pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c6f3233-64d4-424d-9f24-b583d6beacf7-dbus\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864790    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-etc-openvswitch\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-sysconfig\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-cni-netd\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19f50166-5ce1-4963-b621-8366bf342528-ovnkube-script-lib\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-sys-fs\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-log-socket\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.865009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865011    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-device-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.865660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865029    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-cni-dir\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.865660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.864986    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-run-systemd\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.865660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-run-multus-certs\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.865660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-systemd-units\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.865660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-node-log\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.865660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-sys-fs\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.865660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c6f3233-64d4-424d-9f24-b583d6beacf7-kubelet-config\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:41.865660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865537    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-var-lib-kubelet\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.866062 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.866062 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/845662cd-3d72-4f01-bf8f-2d448337f140-tmp\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.866062 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6j6\" (UniqueName: \"kubernetes.io/projected/68af73e0-884e-457e-9e5b-17a3ded1a5c3-kube-api-access-vq6j6\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.866062 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6471e6ef-3d99-4bd7-9cb3-6abca723c71f-iptables-alerter-script\") pod \"iptables-alerter-8gs4b\" (UID: \"6471e6ef-3d99-4bd7-9cb3-6abca723c71f\") " pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.866062 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865867    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68af73e0-884e-457e-9e5b-17a3ded1a5c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.866062 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.865940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-os-release\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.866338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-run-openvswitch\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/68af73e0-884e-457e-9e5b-17a3ded1a5c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.866338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-run-netns\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866230    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4382b057-9b5d-43fb-8d3e-a64b33732a92-host\") pod \"node-ca-986jm\" (UID: \"4382b057-9b5d-43fb-8d3e-a64b33732a92\") " pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.866338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866278    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6471e6ef-3d99-4bd7-9cb3-6abca723c71f-host-slash\") pod \"iptables-alerter-8gs4b\" (UID: \"6471e6ef-3d99-4bd7-9cb3-6abca723c71f\") " pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.866338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19f50166-5ce1-4963-b621-8366bf342528-env-overrides\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866606 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866369    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-system-cni-dir\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.866606 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866454    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c6f3233-64d4-424d-9f24-b583d6beacf7-dbus\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:41.866606 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866504    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866606 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/68af73e0-884e-457e-9e5b-17a3ded1a5c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.866606 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866567    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-run-ovn-kubernetes\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19f50166-5ce1-4963-b621-8366bf342528-ovnkube-config\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/98966b68-fcf4-411b-9db9-20fd9012cd26-hosts-file\") pod \"node-resolver-mtzfw\" (UID: \"98966b68-fcf4-411b-9db9-20fd9012cd26\") " pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knnfw\" (UniqueName: \"kubernetes.io/projected/39014b65-63d6-416a-90bd-9baf56efe171-kube-api-access-knnfw\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-registration-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-run-ovn\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-var-lib-openvswitch\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-cni-bin\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866788    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-kubelet\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.866833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-var-lib-cni-multus\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh4dj\" (UniqueName: \"kubernetes.io/projected/845662cd-3d72-4f01-bf8f-2d448337f140-kube-api-access-qh4dj\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4153a55-9fee-476f-a80a-155d275f14e1-agent-certs\") pod \"konnectivity-agent-2ffrd\" (UID: \"e4153a55-9fee-476f-a80a-155d275f14e1\") " pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98966b68-fcf4-411b-9db9-20fd9012cd26-tmp-dir\") pod \"node-resolver-mtzfw\" (UID: \"98966b68-fcf4-411b-9db9-20fd9012cd26\") " pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.866966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-socket-dir\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867026    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-socket-dir-parent\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4382b057-9b5d-43fb-8d3e-a64b33732a92-serviceca\") pod \"node-ca-986jm\" (UID: \"4382b057-9b5d-43fb-8d3e-a64b33732a92\") " pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-run-ovn\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19f50166-5ce1-4963-b621-8366bf342528-ovn-node-metrics-cert\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-cni-bin\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867088    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-etc-selinux\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-sysctl-d\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-cnibin\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.867178    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867127    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-run-ovn-kubernetes\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.867260    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret podName:1c6f3233-64d4-424d-9f24-b583d6beacf7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:42.367236815 +0000 UTC m=+3.021731781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret") pod "global-pull-secret-syncer-rff9b" (UID: "1c6f3233-64d4-424d-9f24-b583d6beacf7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:41.867448 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867276    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/163783e7-8154-4d7d-abc4-c4804e868dca-etc-selinux\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.868196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68af73e0-884e-457e-9e5b-17a3ded1a5c3-cnibin\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.868196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-cni-netd\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.868196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19f50166-5ce1-4963-b621-8366bf342528-ovnkube-config\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.868196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.867967    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:22:41.868196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.868033    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-host-slash\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.868196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.868065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19f50166-5ce1-4963-b621-8366bf342528-etc-openvswitch\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.868555 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.868298    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98966b68-fcf4-411b-9db9-20fd9012cd26-tmp-dir\") pod \"node-resolver-mtzfw\" (UID: \"98966b68-fcf4-411b-9db9-20fd9012cd26\") " pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.868705 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.868686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19f50166-5ce1-4963-b621-8366bf342528-ovnkube-script-lib\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.872757 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.872717    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19f50166-5ce1-4963-b621-8366bf342528-ovn-node-metrics-cert\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.877641 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.877617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbpn\" (UniqueName: \"kubernetes.io/projected/4382b057-9b5d-43fb-8d3e-a64b33732a92-kube-api-access-pjbpn\") pod \"node-ca-986jm\" (UID: \"4382b057-9b5d-43fb-8d3e-a64b33732a92\") " pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:41.878516 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.878491    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87bl\" (UniqueName: \"kubernetes.io/projected/6471e6ef-3d99-4bd7-9cb3-6abca723c71f-kube-api-access-c87bl\") pod \"iptables-alerter-8gs4b\" (UID: \"6471e6ef-3d99-4bd7-9cb3-6abca723c71f\") " pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:41.878664 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.878545    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6j6\" (UniqueName: \"kubernetes.io/projected/68af73e0-884e-457e-9e5b-17a3ded1a5c3-kube-api-access-vq6j6\") pod \"multus-additional-cni-plugins-dxvgt\" (UID: \"68af73e0-884e-457e-9e5b-17a3ded1a5c3\") " pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:41.879427 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.879400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnkb\" (UniqueName: \"kubernetes.io/projected/98966b68-fcf4-411b-9db9-20fd9012cd26-kube-api-access-tdnkb\") pod \"node-resolver-mtzfw\" (UID: \"98966b68-fcf4-411b-9db9-20fd9012cd26\") " pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:41.880066 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.880043    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk2vg\" (UniqueName: \"kubernetes.io/projected/19f50166-5ce1-4963-b621-8366bf342528-kube-api-access-nk2vg\") pod \"ovnkube-node-nrwj7\" (UID: \"19f50166-5ce1-4963-b621-8366bf342528\") " pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:41.880230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.880210    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46kg\" (UniqueName: \"kubernetes.io/projected/163783e7-8154-4d7d-abc4-c4804e868dca-kube-api-access-f46kg\") pod \"aws-ebs-csi-driver-node-kr8kl\" (UID: \"163783e7-8154-4d7d-abc4-c4804e868dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:41.968324 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968239    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-var-lib-kubelet\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.968324 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/845662cd-3d72-4f01-bf8f-2d448337f140-tmp\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.968324 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knnfw\" (UniqueName: \"kubernetes.io/projected/39014b65-63d6-416a-90bd-9baf56efe171-kube-api-access-knnfw\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-var-lib-cni-multus\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4dj\" (UniqueName: \"kubernetes.io/projected/845662cd-3d72-4f01-bf8f-2d448337f140-kube-api-access-qh4dj\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-var-lib-kubelet\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968390    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4153a55-9fee-476f-a80a-155d275f14e1-agent-certs\") pod \"konnectivity-agent-2ffrd\" (UID: \"e4153a55-9fee-476f-a80a-155d275f14e1\") " pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-socket-dir-parent\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-var-lib-cni-multus\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-sysctl-d\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhq2z\" (UniqueName: \"kubernetes.io/projected/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-kube-api-access-jhq2z\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968512    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-socket-dir-parent\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968537    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-run-k8s-cni-cncf-io\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968565    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-cnibin\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.968580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968587    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-os-release\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968608    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-sysctl-conf\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-host\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-cni-binary-copy\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-run-netns\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-cnibin\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-var-lib-kubelet\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-sysctl-d\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-hostroot\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-run-k8s-cni-cncf-io\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-modprobe-d\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-hostroot\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-lib-modules\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968818    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-os-release\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-system-cni-dir\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968813    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-host\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-etc-kubernetes\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-sys\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.969230 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-run-netns\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-daemon-config\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-kubernetes\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968950    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-system-cni-dir\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-conf-dir\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-systemd\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968989    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-conf-dir\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/845662cd-3d72-4f01-bf8f-2d448337f140-etc-tuned\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.968953    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-sysctl-conf\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-var-lib-cni-bin\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969025    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-etc-kubernetes\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-var-lib-kubelet\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-kubernetes\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-sys\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969181    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-modprobe-d\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-var-lib-cni-bin\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-run\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969261    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-lib-modules\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.970248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969293    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-systemd\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969372    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-cni-binary-copy\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969379    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-run\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4153a55-9fee-476f-a80a-155d275f14e1-konnectivity-ca\") pod \"konnectivity-agent-2ffrd\" (UID: \"e4153a55-9fee-476f-a80a-155d275f14e1\") " pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-sysconfig\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.969659    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.969726    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs podName:39014b65-63d6-416a-90bd-9baf56efe171 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:42.469706554 +0000 UTC m=+3.124201507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs") pod "network-metrics-daemon-89z9z" (UID: "39014b65-63d6-416a-90bd-9baf56efe171") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969841    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-daemon-config\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4153a55-9fee-476f-a80a-155d275f14e1-konnectivity-ca\") pod \"konnectivity-agent-2ffrd\" (UID: \"e4153a55-9fee-476f-a80a-155d275f14e1\") " pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/845662cd-3d72-4f01-bf8f-2d448337f140-etc-sysconfig\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.969949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-cni-dir\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.970004    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-multus-cni-dir\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.970040    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-run-multus-certs\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.971150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.970084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-host-run-multus-certs\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.971765 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.971447    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4153a55-9fee-476f-a80a-155d275f14e1-agent-certs\") pod \"konnectivity-agent-2ffrd\" (UID: \"e4153a55-9fee-476f-a80a-155d275f14e1\") " pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:22:41.971765 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.971473    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/845662cd-3d72-4f01-bf8f-2d448337f140-etc-tuned\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.971824 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.971791    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/845662cd-3d72-4f01-bf8f-2d448337f140-tmp\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:41.976155 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.976132    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:41.976279 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.976158    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:41.976279 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.976173    2559 projected.go:196] Error preparing data for projected volume kube-api-access-x7ptk for pod openshift-network-diagnostics/network-check-target-jzkpr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:41.976279 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:41.976248    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk podName:42477d3f-a75e-48bc-89d5-176f6a162f00 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:42.476232651 +0000 UTC m=+3.130727604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ptk" (UniqueName: "kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk") pod "network-check-target-jzkpr" (UID: "42477d3f-a75e-48bc-89d5-176f6a162f00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:41.978053 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.978038    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhq2z\" (UniqueName: \"kubernetes.io/projected/020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b-kube-api-access-jhq2z\") pod \"multus-zcfck\" (UID: \"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b\") " pod="openshift-multus/multus-zcfck"
Jul 28 17:22:41.980898 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.980849    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knnfw\" (UniqueName: \"kubernetes.io/projected/39014b65-63d6-416a-90bd-9baf56efe171-kube-api-access-knnfw\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:41.981012 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:41.981002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4dj\" (UniqueName: \"kubernetes.io/projected/845662cd-3d72-4f01-bf8f-2d448337f140-kube-api-access-qh4dj\") pod \"tuned-shs4g\" (UID: \"845662cd-3d72-4f01-bf8f-2d448337f140\") " pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:42.070257 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.070219    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl"
Jul 28 17:22:42.077609 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.077585    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dxvgt"
Jul 28 17:22:42.089281 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.089250    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mtzfw"
Jul 28 17:22:42.095951 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.095930    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-986jm"
Jul 28 17:22:42.105591 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.105573    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8gs4b"
Jul 28 17:22:42.119301 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.119278    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:22:42.124071 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.124051    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:22:42.131747 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.131719    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-shs4g"
Jul 28 17:22:42.138446 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.138424    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zcfck"
Jul 28 17:22:42.372089 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.372053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:42.372246 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:42.372208    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:42.372291 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:42.372274    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret podName:1c6f3233-64d4-424d-9f24-b583d6beacf7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:43.372258849 +0000 UTC m=+4.026753803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret") pod "global-pull-secret-syncer-rff9b" (UID: "1c6f3233-64d4-424d-9f24-b583d6beacf7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:42.472819 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.472776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:42.472987 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:42.472936    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:42.473064 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:42.473003    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs podName:39014b65-63d6-416a-90bd-9baf56efe171 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:43.472984866 +0000 UTC m=+4.127479835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs") pod "network-metrics-daemon-89z9z" (UID: "39014b65-63d6-416a-90bd-9baf56efe171") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:42.573967 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.573927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:42.574165 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:42.574094    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:42.574165 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:42.574116    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:42.574165 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:42.574127    2559 projected.go:196] Error preparing data for projected volume kube-api-access-x7ptk for pod openshift-network-diagnostics/network-check-target-jzkpr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:42.574329 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:42.574186    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk podName:42477d3f-a75e-48bc-89d5-176f6a162f00 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:43.574167437 +0000 UTC m=+4.228662392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ptk" (UniqueName: "kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk") pod "network-check-target-jzkpr" (UID: "42477d3f-a75e-48bc-89d5-176f6a162f00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:42.805749 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.805669    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:17:41 +0000 UTC" deadline="2028-03-20 14:19:12.932275782 +0000 UTC"
Jul 28 17:22:42.805749 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.805700    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14420h56m30.126579086s"
Jul 28 17:22:42.827086 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:42.827053    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:42.827227 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:42.827166    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:42.987953 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:42.987925    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163783e7_8154_4d7d_abc4_c4804e868dca.slice/crio-12d8b679dabc1921efdf12dc11a7e308a063133de4cd87c445bb1c1bc8384868 WatchSource:0}: Error finding container 12d8b679dabc1921efdf12dc11a7e308a063133de4cd87c445bb1c1bc8384868: Status 404 returned error can't find the container with id 12d8b679dabc1921efdf12dc11a7e308a063133de4cd87c445bb1c1bc8384868
Jul 28 17:22:42.989287 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:42.989269    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4153a55_9fee_476f_a80a_155d275f14e1.slice/crio-74a8a7e967d3caab92e9237be50e0a5edf2983b409eeaf5ab25c91b2d7e0e63e WatchSource:0}: Error finding container 74a8a7e967d3caab92e9237be50e0a5edf2983b409eeaf5ab25c91b2d7e0e63e: Status 404 returned error can't find the container with id 74a8a7e967d3caab92e9237be50e0a5edf2983b409eeaf5ab25c91b2d7e0e63e
Jul 28 17:22:42.990719 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:42.990697    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68af73e0_884e_457e_9e5b_17a3ded1a5c3.slice/crio-f2459d92051ebfbc289814b2cc8138879e8b0b43d13478a2cedd7e7613b03736 WatchSource:0}: Error finding container f2459d92051ebfbc289814b2cc8138879e8b0b43d13478a2cedd7e7613b03736: Status 404 returned error can't find the container with id f2459d92051ebfbc289814b2cc8138879e8b0b43d13478a2cedd7e7613b03736
Jul 28 17:22:42.993106 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:42.993079    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f50166_5ce1_4963_b621_8366bf342528.slice/crio-433126d2fc20a35ce3dc87fe0824f0ecc83312643e56124eb265bd3d2be8ea43 WatchSource:0}: Error finding container 433126d2fc20a35ce3dc87fe0824f0ecc83312643e56124eb265bd3d2be8ea43: Status 404 returned error can't find the container with id 433126d2fc20a35ce3dc87fe0824f0ecc83312643e56124eb265bd3d2be8ea43
Jul 28 17:22:42.994292 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:42.994273    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020a8f3e_cfe7_4a8e_b390_3b8c67ca4d8b.slice/crio-a1006c7816737c3debbe36428c59a8372de3bde7139a7dd94f01b7730a00d930 WatchSource:0}: Error finding container a1006c7816737c3debbe36428c59a8372de3bde7139a7dd94f01b7730a00d930: Status 404 returned error can't find the container with id a1006c7816737c3debbe36428c59a8372de3bde7139a7dd94f01b7730a00d930
Jul 28 17:22:42.995357 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:42.995247    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6471e6ef_3d99_4bd7_9cb3_6abca723c71f.slice/crio-1c34adde404f453c2c55e62c8933a000029df6b5b222e0047b8dfc4fdd399c87 WatchSource:0}: Error finding container 1c34adde404f453c2c55e62c8933a000029df6b5b222e0047b8dfc4fdd399c87: Status 404 returned error can't find the container with id 1c34adde404f453c2c55e62c8933a000029df6b5b222e0047b8dfc4fdd399c87
Jul 28 17:22:42.997448 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:42.997424    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845662cd_3d72_4f01_bf8f_2d448337f140.slice/crio-35cdc394089fb100fea5f5ad5a7f15c9b286e1af3951b52338ae18743b302b9f WatchSource:0}: Error finding container 35cdc394089fb100fea5f5ad5a7f15c9b286e1af3951b52338ae18743b302b9f: Status 404 returned error can't find the container with id 35cdc394089fb100fea5f5ad5a7f15c9b286e1af3951b52338ae18743b302b9f
Jul 28 17:22:42.997810 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:42.997728    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98966b68_fcf4_411b_9db9_20fd9012cd26.slice/crio-dff910d77e1cd0debf70de8acc254681b1eb40a5033f28de08b5869f3311414e WatchSource:0}: Error finding container dff910d77e1cd0debf70de8acc254681b1eb40a5033f28de08b5869f3311414e: Status 404 returned error can't find the container with id dff910d77e1cd0debf70de8acc254681b1eb40a5033f28de08b5869f3311414e
Jul 28 17:22:42.999342 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:22:42.998674    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4382b057_9b5d_43fb_8d3e_a64b33732a92.slice/crio-98e60f263461ab9c01ec26d91a5b6a4b4cf1d13f5221f8db36167aabf8ca0547 WatchSource:0}: Error finding container 98e60f263461ab9c01ec26d91a5b6a4b4cf1d13f5221f8db36167aabf8ca0547: Status 404 returned error can't find the container with id 98e60f263461ab9c01ec26d91a5b6a4b4cf1d13f5221f8db36167aabf8ca0547
Jul 28 17:22:43.381220 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.381005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:43.381390 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.381182    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:43.381448 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.381420    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret podName:1c6f3233-64d4-424d-9f24-b583d6beacf7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:45.381392147 +0000 UTC m=+6.035887100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret") pod "global-pull-secret-syncer-rff9b" (UID: "1c6f3233-64d4-424d-9f24-b583d6beacf7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:43.482180 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.482140    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:43.482352 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.482331    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:43.482426 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.482395    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs podName:39014b65-63d6-416a-90bd-9baf56efe171 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:45.482375318 +0000 UTC m=+6.136870276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs") pod "network-metrics-daemon-89z9z" (UID: "39014b65-63d6-416a-90bd-9baf56efe171") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:43.582605 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.582564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:43.582797 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.582736    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:43.582797 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.582757    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:43.582797 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.582769    2559 projected.go:196] Error preparing data for projected volume kube-api-access-x7ptk for pod openshift-network-diagnostics/network-check-target-jzkpr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:43.582983 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.582828    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk podName:42477d3f-a75e-48bc-89d5-176f6a162f00 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:45.582810782 +0000 UTC m=+6.237305741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ptk" (UniqueName: "kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk") pod "network-check-target-jzkpr" (UID: "42477d3f-a75e-48bc-89d5-176f6a162f00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:43.808138 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.808030    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:17:41 +0000 UTC" deadline="2028-02-28 05:31:57.203329542 +0000 UTC"
Jul 28 17:22:43.808138 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.808098    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13908h9m13.395236609s"
Jul 28 17:22:43.829905 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.829863    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:43.830024 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.829983    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:43.830494 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.830308    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:43.830494 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:43.830400    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:22:43.872185 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.872148    2559 generic.go:356] "Generic (PLEG): container finished" podID="ab14e603e28189365da9a7adafc8be56" containerID="984da11d543a15409e1826b1b0e398eec8642feeb91e8963a3f2e4a973875c91" exitCode=0
Jul 28 17:22:43.872563 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.872478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal" event={"ID":"ab14e603e28189365da9a7adafc8be56","Type":"ContainerDied","Data":"984da11d543a15409e1826b1b0e398eec8642feeb91e8963a3f2e4a973875c91"}
Jul 28 17:22:43.890054 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.890003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-986jm" event={"ID":"4382b057-9b5d-43fb-8d3e-a64b33732a92","Type":"ContainerStarted","Data":"98e60f263461ab9c01ec26d91a5b6a4b4cf1d13f5221f8db36167aabf8ca0547"}
Jul 28 17:22:43.895920 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.895819    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-shs4g" event={"ID":"845662cd-3d72-4f01-bf8f-2d448337f140","Type":"ContainerStarted","Data":"35cdc394089fb100fea5f5ad5a7f15c9b286e1af3951b52338ae18743b302b9f"}
Jul 28 17:22:43.899236 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.899160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zcfck" event={"ID":"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b","Type":"ContainerStarted","Data":"a1006c7816737c3debbe36428c59a8372de3bde7139a7dd94f01b7730a00d930"}
Jul 28 17:22:43.906169 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.906132    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2ffrd" event={"ID":"e4153a55-9fee-476f-a80a-155d275f14e1","Type":"ContainerStarted","Data":"74a8a7e967d3caab92e9237be50e0a5edf2983b409eeaf5ab25c91b2d7e0e63e"}
Jul 28 17:22:43.910679 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.909930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal" event={"ID":"f12b46d1950955bc6884ff5ac3c8c404","Type":"ContainerStarted","Data":"5a659946c21c9ecb4fa523b82a50671ecad68b0faaf8d1adf7a16e38a75af3cb"}
Jul 28 17:22:43.917807 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.917773    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mtzfw" event={"ID":"98966b68-fcf4-411b-9db9-20fd9012cd26","Type":"ContainerStarted","Data":"dff910d77e1cd0debf70de8acc254681b1eb40a5033f28de08b5869f3311414e"}
Jul 28 17:22:43.921414 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.921387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8gs4b" event={"ID":"6471e6ef-3d99-4bd7-9cb3-6abca723c71f","Type":"ContainerStarted","Data":"1c34adde404f453c2c55e62c8933a000029df6b5b222e0047b8dfc4fdd399c87"}
Jul 28 17:22:43.924549 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.924525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxvgt" event={"ID":"68af73e0-884e-457e-9e5b-17a3ded1a5c3","Type":"ContainerStarted","Data":"f2459d92051ebfbc289814b2cc8138879e8b0b43d13478a2cedd7e7613b03736"}
Jul 28 17:22:43.939533 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.939494    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" event={"ID":"19f50166-5ce1-4963-b621-8366bf342528","Type":"ContainerStarted","Data":"433126d2fc20a35ce3dc87fe0824f0ecc83312643e56124eb265bd3d2be8ea43"}
Jul 28 17:22:43.949901 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:43.947790    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl" event={"ID":"163783e7-8154-4d7d-abc4-c4804e868dca","Type":"ContainerStarted","Data":"12d8b679dabc1921efdf12dc11a7e308a063133de4cd87c445bb1c1bc8384868"}
Jul 28 17:22:44.827409 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:44.827366    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:44.827997 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:44.827506    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:44.971272 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:44.971196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal" event={"ID":"ab14e603e28189365da9a7adafc8be56","Type":"ContainerStarted","Data":"11508ea99a4a08196b852f531a337c5d25ffd2ee8586c8594b3285dbdfca17d7"}
Jul 28 17:22:45.002028 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:45.000829    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal" podStartSLOduration=4.000805072 podStartE2EDuration="4.000805072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:43.925416756 +0000 UTC m=+4.579911734" watchObservedRunningTime="2026-07-28 17:22:45.000805072 +0000 UTC m=+5.655300050"
Jul 28 17:22:45.398478 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:45.398427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:45.398683 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.398626    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:45.398807 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.398693    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret podName:1c6f3233-64d4-424d-9f24-b583d6beacf7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:49.398674685 +0000 UTC m=+10.053169645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret") pod "global-pull-secret-syncer-rff9b" (UID: "1c6f3233-64d4-424d-9f24-b583d6beacf7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:45.499351 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:45.499297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:45.499531 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.499481    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:45.499604 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.499545    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs podName:39014b65-63d6-416a-90bd-9baf56efe171 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:49.499526653 +0000 UTC m=+10.154021629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs") pod "network-metrics-daemon-89z9z" (UID: "39014b65-63d6-416a-90bd-9baf56efe171") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:45.600553 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:45.600510    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:45.600740 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.600715    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:45.600740 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.600739    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:45.600895 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.600752    2559 projected.go:196] Error preparing data for projected volume kube-api-access-x7ptk for pod openshift-network-diagnostics/network-check-target-jzkpr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:45.600895 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.600812    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk podName:42477d3f-a75e-48bc-89d5-176f6a162f00 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:49.600794042 +0000 UTC m=+10.255289017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ptk" (UniqueName: "kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk") pod "network-check-target-jzkpr" (UID: "42477d3f-a75e-48bc-89d5-176f6a162f00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:45.828050 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:45.827967    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:45.828490 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.828115    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:22:45.828652 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:45.828630    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:45.828785 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:45.828762    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:46.827723 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:46.827674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:46.827957 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:46.827821    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:47.829973 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:47.829934    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:47.830408 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:47.830051    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:22:47.830500 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:47.830445    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:47.830607 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:47.830547    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:48.827486 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:48.827449    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:48.827663 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:48.827578    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:49.433903 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:49.433847    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:49.434343 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.433979    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:49.434343 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.434038    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret podName:1c6f3233-64d4-424d-9f24-b583d6beacf7 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:57.434022874 +0000 UTC m=+18.088517830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret") pod "global-pull-secret-syncer-rff9b" (UID: "1c6f3233-64d4-424d-9f24-b583d6beacf7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:49.535001 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:49.534864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:49.535281 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.535008    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:49.535281 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.535081    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs podName:39014b65-63d6-416a-90bd-9baf56efe171 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:57.535060036 +0000 UTC m=+18.189554995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs") pod "network-metrics-daemon-89z9z" (UID: "39014b65-63d6-416a-90bd-9baf56efe171") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:49.635623 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:49.635592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:49.635808 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.635790    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:49.635890 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.635817    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:49.635890 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.635830    2559 projected.go:196] Error preparing data for projected volume kube-api-access-x7ptk for pod openshift-network-diagnostics/network-check-target-jzkpr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:49.635996 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.635905    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk podName:42477d3f-a75e-48bc-89d5-176f6a162f00 nodeName:}" failed. No retries permitted until 2026-07-28 17:22:57.635866695 +0000 UTC m=+18.290361662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ptk" (UniqueName: "kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk") pod "network-check-target-jzkpr" (UID: "42477d3f-a75e-48bc-89d5-176f6a162f00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:49.830003 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:49.829923    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:49.830287 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:49.830264    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:49.830415 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.830379    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:22:49.830844 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:49.830818    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:50.827505 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:50.827464    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:50.827935 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:50.827577    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:51.828006 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:51.827976    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:51.828451 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:51.828102    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:51.828542 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:51.828475    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:51.828595 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:51.828565    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:22:52.827536 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:52.827502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:52.827696 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:52.827614    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:53.827155 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:53.827117    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:53.827508 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:53.827235    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:22:53.827508 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:53.827262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:53.827508 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:53.827354    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:54.827105 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:54.827070    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:54.827287 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:54.827185    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:55.827008 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:55.826974    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:55.827151 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:55.827097    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:22:55.827521 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:55.827503    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:55.827775 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:55.827613    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:56.827795 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:56.827756    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:56.828290 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:56.827891    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:57.500218 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:57.500186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:57.500454 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.500322    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:57.500454 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.500385    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret podName:1c6f3233-64d4-424d-9f24-b583d6beacf7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:13.500367104 +0000 UTC m=+34.154862059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret") pod "global-pull-secret-syncer-rff9b" (UID: "1c6f3233-64d4-424d-9f24-b583d6beacf7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:22:57.601066 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:57.601034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:57.601224 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.601187    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:57.601273 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.601250    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs podName:39014b65-63d6-416a-90bd-9baf56efe171 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:13.601235505 +0000 UTC m=+34.255730458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs") pod "network-metrics-daemon-89z9z" (UID: "39014b65-63d6-416a-90bd-9baf56efe171") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:22:57.701510 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:57.701476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:57.701678 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.701611    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:22:57.701678 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.701624    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:22:57.701678 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.701633    2559 projected.go:196] Error preparing data for projected volume kube-api-access-x7ptk for pod openshift-network-diagnostics/network-check-target-jzkpr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:57.701805 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.701684    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk podName:42477d3f-a75e-48bc-89d5-176f6a162f00 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:13.701669574 +0000 UTC m=+34.356164528 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ptk" (UniqueName: "kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk") pod "network-check-target-jzkpr" (UID: "42477d3f-a75e-48bc-89d5-176f6a162f00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:22:57.827078 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:57.827010    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:57.827217 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:57.827021    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:57.827217 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.827124    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:57.827306 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:57.827228    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:22:58.828027 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:58.827989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:22:58.828505 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:58.828123    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:22:59.828092 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:59.828064    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:22:59.828499 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:59.828235    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:22:59.828499 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:22:59.828291    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:22:59.828499 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:22:59.828408    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:23:00.827141 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:00.827103    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:00.827329 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:00.827240    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:23:01.826974 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:01.826952    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:01.826974 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:01.826967    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:01.827391 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:01.827053    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:23:01.827391 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:01.827132    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:23:02.018189 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.016823    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" event={"ID":"19f50166-5ce1-4963-b621-8366bf342528","Type":"ContainerStarted","Data":"6dd2c77e9d0b177903eb1e230575dc0070d8b9dc248be4e468d293e6daed45a2"}
Jul 28 17:23:02.019683 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.019652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-986jm" event={"ID":"4382b057-9b5d-43fb-8d3e-a64b33732a92","Type":"ContainerStarted","Data":"2110877f478363a0e3ac82b837f059c5198708e98126e78e1a42734cbbc1cd9d"}
Jul 28 17:23:02.023962 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.023215    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-shs4g" event={"ID":"845662cd-3d72-4f01-bf8f-2d448337f140","Type":"ContainerStarted","Data":"8aeb5186bb7bb5275866f9ef6d962e3f1caca59e0c5ca5a448de4c4c503ebd88"}
Jul 28 17:23:02.031841 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.030998    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zcfck" event={"ID":"020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b","Type":"ContainerStarted","Data":"3668d45b6717679dae5d0c4dc1c21b2dc56f8b053310e701cb1540f1c86db942"}
Jul 28 17:23:02.036470 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.036034    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-986jm" podStartSLOduration=3.210279445 podStartE2EDuration="22.036012848s" totalImagesPullingTime="18.825733403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:02.035444126 +0000 UTC m=+22.689939114" watchObservedRunningTime="2026-07-28 17:23:02.036012848 +0000 UTC m=+22.690507824"
Jul 28 17:23:02.036470 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.036129    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-142.ec2.internal" podStartSLOduration=21.036120301 podStartE2EDuration="21.036120301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:45.001999531 +0000 UTC m=+5.656494519" watchObservedRunningTime="2026-07-28 17:23:02.036120301 +0000 UTC m=+22.690615275"
Jul 28 17:23:02.037618 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.037559    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2ffrd" event={"ID":"e4153a55-9fee-476f-a80a-155d275f14e1","Type":"ContainerStarted","Data":"bf2ff1f334958b653a1e6197295bcf06f10f5cee5e093b0c55555d534de04f0b"}
Jul 28 17:23:02.055706 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.055596    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-shs4g" podStartSLOduration=3.1961787 podStartE2EDuration="22.055575871s" totalImagesPullingTime="18.859397171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:02.055358804 +0000 UTC m=+22.709853780" watchObservedRunningTime="2026-07-28 17:23:02.055575871 +0000 UTC m=+22.710070846"
Jul 28 17:23:02.079920 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.079493    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zcfck" podStartSLOduration=3.198277805 podStartE2EDuration="22.079476655s" totalImagesPullingTime="18.88119885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:02.079417509 +0000 UTC m=+22.733912485" watchObservedRunningTime="2026-07-28 17:23:02.079476655 +0000 UTC m=+22.733971629"
Jul 28 17:23:02.107610 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.107557    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2ffrd" podStartSLOduration=3.295069001 podStartE2EDuration="22.107539863s" totalImagesPullingTime="18.812470862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:02.106850543 +0000 UTC m=+22.761345544" watchObservedRunningTime="2026-07-28 17:23:02.107539863 +0000 UTC m=+22.762034839"
Jul 28 17:23:02.828080 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:02.827806    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:02.829045 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:02.828198    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:23:03.045455 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.045418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" event={"ID":"19f50166-5ce1-4963-b621-8366bf342528","Type":"ContainerStarted","Data":"7f4507d85a7f0960a60b8fccee1d4300ac688dd78c2c24b2e4ec5f51362b0986"}
Jul 28 17:23:03.045640 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.045520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" event={"ID":"19f50166-5ce1-4963-b621-8366bf342528","Type":"ContainerStarted","Data":"bbbc3dd8ebb09471e0156489606fd7c77432a7595771a806088cfbb581ca9662"}
Jul 28 17:23:03.045640 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.045551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" event={"ID":"19f50166-5ce1-4963-b621-8366bf342528","Type":"ContainerStarted","Data":"012e5c4c4e3bd246602751c9380295481e8aa2357ee2c4c2da3e0cfb6e39f88d"}
Jul 28 17:23:03.045640 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.045585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" event={"ID":"19f50166-5ce1-4963-b621-8366bf342528","Type":"ContainerStarted","Data":"d20834315aef3edb71c0ce1e6102dddef093327ae3146bf865c4a3296546148e"}
Jul 28 17:23:03.045640 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.045616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" event={"ID":"19f50166-5ce1-4963-b621-8366bf342528","Type":"ContainerStarted","Data":"367eeda5edd297f9fee7e7d7441f8030376e5240d0a0f1f9255a4edfcc4ea0f0"}
Jul 28 17:23:03.046815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.046792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl" event={"ID":"163783e7-8154-4d7d-abc4-c4804e868dca","Type":"ContainerStarted","Data":"e4ad05fde394ed93138d7f19aecaedc96fbe67e73cf4d76e955ea0b98c6f0f36"}
Jul 28 17:23:03.048233 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.048203    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mtzfw" event={"ID":"98966b68-fcf4-411b-9db9-20fd9012cd26","Type":"ContainerStarted","Data":"119b09820eb3dab656cd24916edfdb821ca2b2d494ed2f6d2ab4faa690b88491"}
Jul 28 17:23:03.049688 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.049664    2559 generic.go:356] "Generic (PLEG): container finished" podID="68af73e0-884e-457e-9e5b-17a3ded1a5c3" containerID="94a5da62aa9e67d0c9bea91a1ad9cdfbf6cbba0fdc2ec200144bcc3b1882bf94" exitCode=0
Jul 28 17:23:03.049782 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.049687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxvgt" event={"ID":"68af73e0-884e-457e-9e5b-17a3ded1a5c3","Type":"ContainerDied","Data":"94a5da62aa9e67d0c9bea91a1ad9cdfbf6cbba0fdc2ec200144bcc3b1882bf94"}
Jul 28 17:23:03.078691 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.078595    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mtzfw" podStartSLOduration=4.2451352589999996 podStartE2EDuration="23.078578474s" totalImagesPullingTime="18.833443215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:03.078472502 +0000 UTC m=+23.732967566" watchObservedRunningTime="2026-07-28 17:23:03.078578474 +0000 UTC m=+23.733073450"
Jul 28 17:23:03.306842 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.306799    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:23:03.797372 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.797334    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:23:03.827885 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.827742    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:23:03.306823206Z","UUID":"a4d780f5-b06e-4817-833a-739bb0e258f9","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:23:03.828057 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.827908    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:03.828057 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:03.828012    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:23:03.828478 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.828066    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:03.828478 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:03.828213    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:23:03.829558 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.829540    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:23:03.829662 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:03.829580    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:23:04.053799 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:04.053699    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8gs4b" event={"ID":"6471e6ef-3d99-4bd7-9cb3-6abca723c71f","Type":"ContainerStarted","Data":"bee78574c3b9d094c89299a3d725a9f38661fe32722e4dc07aa2e6b1565a3055"}
Jul 28 17:23:04.055706 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:04.055678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl" event={"ID":"163783e7-8154-4d7d-abc4-c4804e868dca","Type":"ContainerStarted","Data":"c3bc777d530d94c37e3efe38eaacead97653f0900b58d50e3160ad382370dae1"}
Jul 28 17:23:04.071393 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:04.071348    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8gs4b" podStartSLOduration=5.244638194 podStartE2EDuration="24.07133283s" totalImagesPullingTime="18.826694636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:04.071304311 +0000 UTC m=+24.725799288" watchObservedRunningTime="2026-07-28 17:23:04.07133283 +0000 UTC m=+24.725827805"
Jul 28 17:23:04.827570 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:04.827496    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:04.827735 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:04.827614    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:23:05.061905 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:05.061784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" event={"ID":"19f50166-5ce1-4963-b621-8366bf342528","Type":"ContainerStarted","Data":"9570fd2fadc0e2259ecea2195100af35f7f04f9601d02ae69551ae26543524ad"}
Jul 28 17:23:05.064222 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:05.064193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl" event={"ID":"163783e7-8154-4d7d-abc4-c4804e868dca","Type":"ContainerStarted","Data":"c614a1717d9a79aef2cc2b5fced5a7ac54445936071455f88097cf23d449669c"}
Jul 28 17:23:05.084051 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:05.083962    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kr8kl" podStartSLOduration=4.969059556 podStartE2EDuration="26.083945742s" totalImagesPullingTime="21.114886186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:05.083782117 +0000 UTC m=+25.738277093" watchObservedRunningTime="2026-07-28 17:23:05.083945742 +0000 UTC m=+25.738440711"
Jul 28 17:23:05.386989 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:05.386957    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:23:05.387734 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:05.387713    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:23:05.828056 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:05.827978    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:05.828226 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:05.827979    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:05.828226 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:05.828094    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:23:05.828226 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:05.828151    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:23:06.067147 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:06.067117    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2ffrd"
Jul 28 17:23:06.827658 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:06.827472    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:06.827851 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:06.827741    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:23:07.828014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:07.827926    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:07.828014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:07.827964    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:07.828811 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:07.828063    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:23:07.828811 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:07.828181    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:23:08.073072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:08.072833    2559 generic.go:356] "Generic (PLEG): container finished" podID="68af73e0-884e-457e-9e5b-17a3ded1a5c3" containerID="e5a515bc104aafc805f602a5d1d636a777203b695346e46c592fea42cdd8e680" exitCode=0
Jul 28 17:23:08.073261 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:08.072924    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxvgt" event={"ID":"68af73e0-884e-457e-9e5b-17a3ded1a5c3","Type":"ContainerDied","Data":"e5a515bc104aafc805f602a5d1d636a777203b695346e46c592fea42cdd8e680"}
Jul 28 17:23:08.080227 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:08.080141    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" event={"ID":"19f50166-5ce1-4963-b621-8366bf342528","Type":"ContainerStarted","Data":"7f1cf0d04ad8552c9457b46993064a7a950951e1fd9274a37a496598e67f76e2"}
Jul 28 17:23:08.080388 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:08.080375    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:23:08.080458 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:08.080434    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:23:08.097081 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:08.097052    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:23:08.131620 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:08.131570    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7" podStartSLOduration=9.232292685 podStartE2EDuration="28.131540236s" totalImagesPullingTime="18.899247551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:08.131331502 +0000 UTC m=+28.785826477" watchObservedRunningTime="2026-07-28 17:23:08.131540236 +0000 UTC m=+28.786035212"
Jul 28 17:23:08.827780 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:08.827748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:08.828030 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:08.827902    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:23:09.082792 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:09.082736    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:23:09.098542 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:09.098516    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:23:09.448583 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:09.448538    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-89z9z"]
Jul 28 17:23:09.448770 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:09.448668    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:09.448841 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:09.448769    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:23:09.451861 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:09.451831    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rff9b"]
Jul 28 17:23:09.452008 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:09.451946    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:09.452075 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:09.452032    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:23:09.452509 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:09.452483    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jzkpr"]
Jul 28 17:23:09.452614 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:09.452586    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:09.452733 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:09.452713    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:23:10.086677 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:10.086643    2559 generic.go:356] "Generic (PLEG): container finished" podID="68af73e0-884e-457e-9e5b-17a3ded1a5c3" containerID="6a7787dcf81acebff5a24ca465e37a962e23982dacfcf681bfdb04db8dd82b1a" exitCode=0
Jul 28 17:23:10.087074 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:10.086739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxvgt" event={"ID":"68af73e0-884e-457e-9e5b-17a3ded1a5c3","Type":"ContainerDied","Data":"6a7787dcf81acebff5a24ca465e37a962e23982dacfcf681bfdb04db8dd82b1a"}
Jul 28 17:23:10.827927 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:10.827898    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:10.828072 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:10.828035    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:23:10.828137 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:10.828120    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:10.828230 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:10.828207    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:23:11.827418 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:11.827338    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:11.827760 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:11.827453    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:23:12.092921 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:12.092823    2559 generic.go:356] "Generic (PLEG): container finished" podID="68af73e0-884e-457e-9e5b-17a3ded1a5c3" containerID="c56395526cdbd0384dcb9348901a53214a4207e6c0d817b68e36e85661ee20d1" exitCode=0
Jul 28 17:23:12.092921 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:12.092893    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxvgt" event={"ID":"68af73e0-884e-457e-9e5b-17a3ded1a5c3","Type":"ContainerDied","Data":"c56395526cdbd0384dcb9348901a53214a4207e6c0d817b68e36e85661ee20d1"}
Jul 28 17:23:12.827280 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:12.827242    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:12.827280 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:12.827268    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:12.827770 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:12.827393    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-89z9z" podUID="39014b65-63d6-416a-90bd-9baf56efe171"
Jul 28 17:23:12.827770 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:12.827525    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzkpr" podUID="42477d3f-a75e-48bc-89d5-176f6a162f00"
Jul 28 17:23:13.515196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:13.515156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:13.515360 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:13.515338    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:13.515434 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:13.515423    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret podName:1c6f3233-64d4-424d-9f24-b583d6beacf7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:45.515400527 +0000 UTC m=+66.169895484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret") pod "global-pull-secret-syncer-rff9b" (UID: "1c6f3233-64d4-424d-9f24-b583d6beacf7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:13.615829 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:13.615793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:13.616021 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:13.616007    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:13.616091 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:13.616078    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs podName:39014b65-63d6-416a-90bd-9baf56efe171 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:45.616057997 +0000 UTC m=+66.270552952 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs") pod "network-metrics-daemon-89z9z" (UID: "39014b65-63d6-416a-90bd-9baf56efe171") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:13.717121 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:13.717090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:13.717268 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:13.717220    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:23:13.717268 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:13.717235    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:23:13.717268 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:13.717244    2559 projected.go:196] Error preparing data for projected volume kube-api-access-x7ptk for pod openshift-network-diagnostics/network-check-target-jzkpr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:13.717400 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:13.717295    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk podName:42477d3f-a75e-48bc-89d5-176f6a162f00 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:45.717279947 +0000 UTC m=+66.371774900 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x7ptk" (UniqueName: "kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk") pod "network-check-target-jzkpr" (UID: "42477d3f-a75e-48bc-89d5-176f6a162f00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:13.827901 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:13.827795    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:13.828331 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:13.827953    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rff9b" podUID="1c6f3233-64d4-424d-9f24-b583d6beacf7"
Jul 28 17:23:14.480922 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.480891    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-142.ec2.internal" event="NodeReady"
Jul 28 17:23:14.481108 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.481053    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:23:14.523537 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.523497    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bb87c65cd-cqhwp"]
Jul 28 17:23:14.528274 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.528243    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.528416 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.528142    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"]
Jul 28 17:23:14.531690 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.531434    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:23:14.531690 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.531471    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:23:14.531690 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.531493    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:23:14.531690 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.531501    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d8jtm\""
Jul 28 17:23:14.532002 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.531809    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"]
Jul 28 17:23:14.532002 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.531913    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.534562 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.534539    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 17:23:14.534686 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.534544    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:23:14.535051 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.535025    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"]
Jul 28 17:23:14.535146 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.535058    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 17:23:14.535146 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.535120    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:23:14.535382 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.535362    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 17:23:14.535653 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.535634    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 17:23:14.535653 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.535647    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.536392 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.536139    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:23:14.537295 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.537274    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:23:14.537839 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.537815    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 17:23:14.538284 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.538265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"
Jul 28 17:23:14.540538 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.540518    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vq4tf\""
Jul 28 17:23:14.541939 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.541857    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 17:23:14.542491 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.542475    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-496j7"]
Jul 28 17:23:14.547966 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.547946    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6wjcw"]
Jul 28 17:23:14.548068 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.548051    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.550271 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.550239    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:23:14.550492 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.550476    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gw5z2\""
Jul 28 17:23:14.551000 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.550643    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:23:14.552037 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.552017    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"]
Jul 28 17:23:14.552298 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.552056    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"]
Jul 28 17:23:14.552298 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.552073    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"]
Jul 28 17:23:14.552298 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.552089    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bb87c65cd-cqhwp"]
Jul 28 17:23:14.552298 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.552133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6wjcw"
Jul 28 17:23:14.552551 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.552373    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6wjcw"]
Jul 28 17:23:14.554372 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.554351    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:23:14.554480 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.554395    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m7mlr\""
Jul 28 17:23:14.554480 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.554404    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:23:14.554480 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.554454    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:23:14.560203 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.560185    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-496j7"]
Jul 28 17:23:14.622586 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.622555    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-bound-sa-token\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.622710 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.622607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5nm\" (UniqueName: \"kubernetes.io/projected/30c1a0b6-194c-4bd5-be83-b9ce463e176b-kube-api-access-7q5nm\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.622832 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.622639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97dd18ff-36f0-45f4-8b65-0fe85052c2c4-tmp\") pod \"klusterlet-addon-workmgr-677d6f476c-s9fmn\" (UID: \"97dd18ff-36f0-45f4-8b65-0fe85052c2c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.623072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-ca\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.623147 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.623272 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.623335 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-image-registry-private-configuration\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.623335 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-registry-certificates\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.623529 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623499    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c1a0b6-194c-4bd5-be83-b9ce463e176b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.623634 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623549    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-trusted-ca\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.623634 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwr6r\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-kube-api-access-nwr6r\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.623634 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-hub\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.623786 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.623786 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/97dd18ff-36f0-45f4-8b65-0fe85052c2c4-klusterlet-config\") pod \"klusterlet-addon-workmgr-677d6f476c-s9fmn\" (UID: \"97dd18ff-36f0-45f4-8b65-0fe85052c2c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.623786 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-installation-pull-secrets\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.623786 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623737    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9t55\" (UniqueName: \"kubernetes.io/projected/97dd18ff-36f0-45f4-8b65-0fe85052c2c4-kube-api-access-t9t55\") pod \"klusterlet-addon-workmgr-677d6f476c-s9fmn\" (UID: \"97dd18ff-36f0-45f4-8b65-0fe85052c2c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.623786 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623759    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87208fe0-5665-4ff9-8a45-78900fa38288-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9\" (UID: \"87208fe0-5665-4ff9-8a45-78900fa38288\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"
Jul 28 17:23:14.623786 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623775    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvj2g\" (UniqueName: \"kubernetes.io/projected/87208fe0-5665-4ff9-8a45-78900fa38288-kube-api-access-gvj2g\") pod \"managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9\" (UID: \"87208fe0-5665-4ff9-8a45-78900fa38288\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"
Jul 28 17:23:14.624075 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.623805    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7890b5d6-939f-420e-bfd1-73793793f024-ca-trust-extracted\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.724826 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.724792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-hub\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.724826 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.724829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.725071 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.724849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/97dd18ff-36f0-45f4-8b65-0fe85052c2c4-klusterlet-config\") pod \"klusterlet-addon-workmgr-677d6f476c-s9fmn\" (UID: \"97dd18ff-36f0-45f4-8b65-0fe85052c2c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.725071 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.724927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.725071 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.724981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-installation-pull-secrets\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.725071 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9t55\" (UniqueName: \"kubernetes.io/projected/97dd18ff-36f0-45f4-8b65-0fe85052c2c4-kube-api-access-t9t55\") pod \"klusterlet-addon-workmgr-677d6f476c-s9fmn\" (UID: \"97dd18ff-36f0-45f4-8b65-0fe85052c2c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.725071 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87208fe0-5665-4ff9-8a45-78900fa38288-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9\" (UID: \"87208fe0-5665-4ff9-8a45-78900fa38288\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"
Jul 28 17:23:14.725316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvj2g\" (UniqueName: \"kubernetes.io/projected/87208fe0-5665-4ff9-8a45-78900fa38288-kube-api-access-gvj2g\") pod \"managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9\" (UID: \"87208fe0-5665-4ff9-8a45-78900fa38288\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"
Jul 28 17:23:14.725316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725121    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5edec3e6-cd2b-40b0-a939-e91932b835c8-cert\") pod \"ingress-canary-6wjcw\" (UID: \"5edec3e6-cd2b-40b0-a939-e91932b835c8\") " pod="openshift-ingress-canary/ingress-canary-6wjcw"
Jul 28 17:23:14.725316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9410f0ea-7664-4c08-986d-7950cfcabfc7-config-volume\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.725316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7890b5d6-939f-420e-bfd1-73793793f024-ca-trust-extracted\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.725316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725203    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt4tl\" (UniqueName: \"kubernetes.io/projected/9410f0ea-7664-4c08-986d-7950cfcabfc7-kube-api-access-mt4tl\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.725316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-bound-sa-token\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.725316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4shs\" (UniqueName: \"kubernetes.io/projected/5edec3e6-cd2b-40b0-a939-e91932b835c8-kube-api-access-q4shs\") pod \"ingress-canary-6wjcw\" (UID: \"5edec3e6-cd2b-40b0-a939-e91932b835c8\") " pod="openshift-ingress-canary/ingress-canary-6wjcw"
Jul 28 17:23:14.725316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5nm\" (UniqueName: \"kubernetes.io/projected/30c1a0b6-194c-4bd5-be83-b9ce463e176b-kube-api-access-7q5nm\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725321    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9410f0ea-7664-4c08-986d-7950cfcabfc7-tmp-dir\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725353    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97dd18ff-36f0-45f4-8b65-0fe85052c2c4-tmp\") pod \"klusterlet-addon-workmgr-677d6f476c-s9fmn\" (UID: \"97dd18ff-36f0-45f4-8b65-0fe85052c2c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-ca\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-image-registry-private-configuration\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-registry-certificates\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c1a0b6-194c-4bd5-be83-b9ce463e176b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-trusted-ca\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.725673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwr6r\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-kube-api-access-nwr6r\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.726140 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.725713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7890b5d6-939f-420e-bfd1-73793793f024-ca-trust-extracted\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.727090 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.726966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c1a0b6-194c-4bd5-be83-b9ce463e176b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.727630 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.727600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-registry-certificates\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.728267 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.728237    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-trusted-ca\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.728349 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.728285    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97dd18ff-36f0-45f4-8b65-0fe85052c2c4-tmp\") pod \"klusterlet-addon-workmgr-677d6f476c-s9fmn\" (UID: \"97dd18ff-36f0-45f4-8b65-0fe85052c2c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.728419 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:14.728372    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:23:14.728419 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:14.728387    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb87c65cd-cqhwp: secret "image-registry-tls" not found
Jul 28 17:23:14.728992 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:14.728972    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls podName:7890b5d6-939f-420e-bfd1-73793793f024 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:15.22894792 +0000 UTC m=+35.883442873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls") pod "image-registry-5bb87c65cd-cqhwp" (UID: "7890b5d6-939f-420e-bfd1-73793793f024") : secret "image-registry-tls" not found
Jul 28 17:23:14.730306 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.730263    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.730986 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.730960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-installation-pull-secrets\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.731069 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.730997    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/97dd18ff-36f0-45f4-8b65-0fe85052c2c4-klusterlet-config\") pod \"klusterlet-addon-workmgr-677d6f476c-s9fmn\" (UID: \"97dd18ff-36f0-45f4-8b65-0fe85052c2c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.731069 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.731022    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87208fe0-5665-4ff9-8a45-78900fa38288-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9\" (UID: \"87208fe0-5665-4ff9-8a45-78900fa38288\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"
Jul 28 17:23:14.731069 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.731036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.731414 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.731391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-image-registry-private-configuration\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.731528 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.731506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-hub\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.732072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.731971    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/30c1a0b6-194c-4bd5-be83-b9ce463e176b-ca\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.734749 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.734727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9t55\" (UniqueName: \"kubernetes.io/projected/97dd18ff-36f0-45f4-8b65-0fe85052c2c4-kube-api-access-t9t55\") pod \"klusterlet-addon-workmgr-677d6f476c-s9fmn\" (UID: \"97dd18ff-36f0-45f4-8b65-0fe85052c2c4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.738567 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.738549    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvj2g\" (UniqueName: \"kubernetes.io/projected/87208fe0-5665-4ff9-8a45-78900fa38288-kube-api-access-gvj2g\") pod \"managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9\" (UID: \"87208fe0-5665-4ff9-8a45-78900fa38288\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"
Jul 28 17:23:14.738681 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.738659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5nm\" (UniqueName: \"kubernetes.io/projected/30c1a0b6-194c-4bd5-be83-b9ce463e176b-kube-api-access-7q5nm\") pod \"cluster-proxy-proxy-agent-cdcf56496-k5f9h\" (UID: \"30c1a0b6-194c-4bd5-be83-b9ce463e176b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.739037 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.739016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-bound-sa-token\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.740093 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.740071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwr6r\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-kube-api-access-nwr6r\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:14.826094 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.826059    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9410f0ea-7664-4c08-986d-7950cfcabfc7-tmp-dir\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.826262 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.826167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.826262 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.826211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5edec3e6-cd2b-40b0-a939-e91932b835c8-cert\") pod \"ingress-canary-6wjcw\" (UID: \"5edec3e6-cd2b-40b0-a939-e91932b835c8\") " pod="openshift-ingress-canary/ingress-canary-6wjcw"
Jul 28 17:23:14.826262 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.826235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9410f0ea-7664-4c08-986d-7950cfcabfc7-config-volume\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.826372 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.826263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt4tl\" (UniqueName: \"kubernetes.io/projected/9410f0ea-7664-4c08-986d-7950cfcabfc7-kube-api-access-mt4tl\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.826372 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.826289    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4shs\" (UniqueName: \"kubernetes.io/projected/5edec3e6-cd2b-40b0-a939-e91932b835c8-kube-api-access-q4shs\") pod \"ingress-canary-6wjcw\" (UID: \"5edec3e6-cd2b-40b0-a939-e91932b835c8\") " pod="openshift-ingress-canary/ingress-canary-6wjcw"
Jul 28 17:23:14.826982 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.826939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9410f0ea-7664-4c08-986d-7950cfcabfc7-tmp-dir\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.827117 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:14.827008    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:14.827185 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:14.827126    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls podName:9410f0ea-7664-4c08-986d-7950cfcabfc7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:15.327103632 +0000 UTC m=+35.981598591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls") pod "dns-default-496j7" (UID: "9410f0ea-7664-4c08-986d-7950cfcabfc7") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:14.827185 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.827167    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:14.827185 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.827175    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:14.827486 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.827459    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9410f0ea-7664-4c08-986d-7950cfcabfc7-config-volume\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.829296 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.829276    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:23:14.829296 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.829288    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rqpml\""
Jul 28 17:23:14.829713 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.829283    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:23:14.829713 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.829408    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dpv4x\""
Jul 28 17:23:14.829713 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.829499    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:23:14.830052 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.830020    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5edec3e6-cd2b-40b0-a939-e91932b835c8-cert\") pod \"ingress-canary-6wjcw\" (UID: \"5edec3e6-cd2b-40b0-a939-e91932b835c8\") " pod="openshift-ingress-canary/ingress-canary-6wjcw"
Jul 28 17:23:14.835304 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.835281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4shs\" (UniqueName: \"kubernetes.io/projected/5edec3e6-cd2b-40b0-a939-e91932b835c8-kube-api-access-q4shs\") pod \"ingress-canary-6wjcw\" (UID: \"5edec3e6-cd2b-40b0-a939-e91932b835c8\") " pod="openshift-ingress-canary/ingress-canary-6wjcw"
Jul 28 17:23:14.835497 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.835477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt4tl\" (UniqueName: \"kubernetes.io/projected/9410f0ea-7664-4c08-986d-7950cfcabfc7-kube-api-access-mt4tl\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:14.861704 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.861678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:23:14.868294 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.868274    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:14.887929 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.887904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"
Jul 28 17:23:14.898774 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:14.898756    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6wjcw"
Jul 28 17:23:15.231012 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:15.230976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:15.231187 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:15.231166    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:23:15.231236 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:15.231193    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb87c65cd-cqhwp: secret "image-registry-tls" not found
Jul 28 17:23:15.231276 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:15.231264    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls podName:7890b5d6-939f-420e-bfd1-73793793f024 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:16.231242118 +0000 UTC m=+36.885737085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls") pod "image-registry-5bb87c65cd-cqhwp" (UID: "7890b5d6-939f-420e-bfd1-73793793f024") : secret "image-registry-tls" not found
Jul 28 17:23:15.331498 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:15.331460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:15.331659 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:15.331640    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:15.331729 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:15.331712    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls podName:9410f0ea-7664-4c08-986d-7950cfcabfc7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:16.331691534 +0000 UTC m=+36.986186497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls") pod "dns-default-496j7" (UID: "9410f0ea-7664-4c08-986d-7950cfcabfc7") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:15.827650 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:15.827615    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:15.830389 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:15.830193    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:23:16.240290 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:16.240050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:16.240407 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:16.240151    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:23:16.240407 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:16.240364    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb87c65cd-cqhwp: secret "image-registry-tls" not found
Jul 28 17:23:16.240493 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:16.240460    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls podName:7890b5d6-939f-420e-bfd1-73793793f024 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:18.240439588 +0000 UTC m=+38.894934556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls") pod "image-registry-5bb87c65cd-cqhwp" (UID: "7890b5d6-939f-420e-bfd1-73793793f024") : secret "image-registry-tls" not found
Jul 28 17:23:16.253523 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:16.253482    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9"]
Jul 28 17:23:16.254223 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:16.254202    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"]
Jul 28 17:23:16.256607 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:16.256539    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"]
Jul 28 17:23:16.258149 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:23:16.258099    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87208fe0_5665_4ff9_8a45_78900fa38288.slice/crio-5fa5877fd1e09e6beaa4ad0032214c4272b1e75fba18944e91a5430c8af40d96 WatchSource:0}: Error finding container 5fa5877fd1e09e6beaa4ad0032214c4272b1e75fba18944e91a5430c8af40d96: Status 404 returned error can't find the container with id 5fa5877fd1e09e6beaa4ad0032214c4272b1e75fba18944e91a5430c8af40d96
Jul 28 17:23:16.259142 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:23:16.259118    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30c1a0b6_194c_4bd5_be83_b9ce463e176b.slice/crio-7c02cffa52ededaee54e34dee820e094d366061f9f767f54a60139d3c119e850 WatchSource:0}: Error finding container 7c02cffa52ededaee54e34dee820e094d366061f9f767f54a60139d3c119e850: Status 404 returned error can't find the container with id 7c02cffa52ededaee54e34dee820e094d366061f9f767f54a60139d3c119e850
Jul 28 17:23:16.262806 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:23:16.262768    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dd18ff_36f0_45f4_8b65_0fe85052c2c4.slice/crio-5b64dbf7c16f85abeb4301299532a8c817286c76f17859c22f4d149cda4b86de WatchSource:0}: Error finding container 5b64dbf7c16f85abeb4301299532a8c817286c76f17859c22f4d149cda4b86de: Status 404 returned error can't find the container with id 5b64dbf7c16f85abeb4301299532a8c817286c76f17859c22f4d149cda4b86de
Jul 28 17:23:16.264200 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:16.264164    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6wjcw"]
Jul 28 17:23:16.268110 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:23:16.268086    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edec3e6_cd2b_40b0_a939_e91932b835c8.slice/crio-84081669d0f796004e50013fd0daec8a2909ae4c385aa93dbd2c1a821c7d9a53 WatchSource:0}: Error finding container 84081669d0f796004e50013fd0daec8a2909ae4c385aa93dbd2c1a821c7d9a53: Status 404 returned error can't find the container with id 84081669d0f796004e50013fd0daec8a2909ae4c385aa93dbd2c1a821c7d9a53
Jul 28 17:23:16.341422 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:16.341392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:16.341634 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:16.341566    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:16.341698 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:16.341643    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls podName:9410f0ea-7664-4c08-986d-7950cfcabfc7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:18.34162288 +0000 UTC m=+38.996117846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls") pod "dns-default-496j7" (UID: "9410f0ea-7664-4c08-986d-7950cfcabfc7") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:16.381154 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:16.381111    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:16.489775 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:16.489740    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:17.109388 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.109345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn" event={"ID":"97dd18ff-36f0-45f4-8b65-0fe85052c2c4","Type":"ContainerStarted","Data":"5b64dbf7c16f85abeb4301299532a8c817286c76f17859c22f4d149cda4b86de"}
Jul 28 17:23:17.111370 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.111306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" event={"ID":"30c1a0b6-194c-4bd5-be83-b9ce463e176b","Type":"ContainerStarted","Data":"7c02cffa52ededaee54e34dee820e094d366061f9f767f54a60139d3c119e850"}
Jul 28 17:23:17.116854 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.116820    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6wjcw" event={"ID":"5edec3e6-cd2b-40b0-a939-e91932b835c8","Type":"ContainerStarted","Data":"84081669d0f796004e50013fd0daec8a2909ae4c385aa93dbd2c1a821c7d9a53"}
Jul 28 17:23:17.120214 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.120177    2559 generic.go:356] "Generic (PLEG): container finished" podID="68af73e0-884e-457e-9e5b-17a3ded1a5c3" containerID="22250f66c932ad4f09adae991487cdaa22793dab2344fc4cb01e716c6ad91075" exitCode=0
Jul 28 17:23:17.120362 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.120284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxvgt" event={"ID":"68af73e0-884e-457e-9e5b-17a3ded1a5c3","Type":"ContainerDied","Data":"22250f66c932ad4f09adae991487cdaa22793dab2344fc4cb01e716c6ad91075"}
Jul 28 17:23:17.122491 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.122450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9" event={"ID":"87208fe0-5665-4ff9-8a45-78900fa38288","Type":"ContainerStarted","Data":"5fa5877fd1e09e6beaa4ad0032214c4272b1e75fba18944e91a5430c8af40d96"}
Jul 28 17:23:17.177172 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.177091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:17.450553 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.450487    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:17.519469 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.519436    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:17.624469 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.624412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:17.729151 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.729049    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:17.827662 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:17.827591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:18.135346 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:18.134429    2559 generic.go:356] "Generic (PLEG): container finished" podID="68af73e0-884e-457e-9e5b-17a3ded1a5c3" containerID="3a66d4d164c563b0da15f1cb03282977fc41909688a389a290ea1114d6418619" exitCode=0
Jul 28 17:23:18.135346 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:18.134505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxvgt" event={"ID":"68af73e0-884e-457e-9e5b-17a3ded1a5c3","Type":"ContainerDied","Data":"3a66d4d164c563b0da15f1cb03282977fc41909688a389a290ea1114d6418619"}
Jul 28 17:23:18.261719 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:18.259925    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:18.261719 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:18.260845    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:23:18.261719 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:18.260863    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb87c65cd-cqhwp: secret "image-registry-tls" not found
Jul 28 17:23:18.261719 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:18.260938    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls podName:7890b5d6-939f-420e-bfd1-73793793f024 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:22.260918127 +0000 UTC m=+42.915413083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls") pod "image-registry-5bb87c65cd-cqhwp" (UID: "7890b5d6-939f-420e-bfd1-73793793f024") : secret "image-registry-tls" not found
Jul 28 17:23:18.361032 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:18.360460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:18.361032 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:18.360647    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:18.361032 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:18.360696    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls podName:9410f0ea-7664-4c08-986d-7950cfcabfc7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:22.360683192 +0000 UTC m=+43.015178145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls") pod "dns-default-496j7" (UID: "9410f0ea-7664-4c08-986d-7950cfcabfc7") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:18.493604 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:18.493518    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:18.596420 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:18.596388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:18.825319 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:18.825188    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:20.258290 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:20.258253    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:22.295949 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:22.295917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:22.296307 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:22.296082    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:23:22.296307 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:22.296098    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb87c65cd-cqhwp: secret "image-registry-tls" not found
Jul 28 17:23:22.296307 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:22.296157    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls podName:7890b5d6-939f-420e-bfd1-73793793f024 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:30.296139756 +0000 UTC m=+50.950634710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls") pod "image-registry-5bb87c65cd-cqhwp" (UID: "7890b5d6-939f-420e-bfd1-73793793f024") : secret "image-registry-tls" not found
Jul 28 17:23:22.397249 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:22.397210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:22.397386 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:22.397336    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:22.397386 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:22.397387    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls podName:9410f0ea-7664-4c08-986d-7950cfcabfc7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:30.397374533 +0000 UTC m=+51.051869497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls") pod "dns-default-496j7" (UID: "9410f0ea-7664-4c08-986d-7950cfcabfc7") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:25.158254 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.158220    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn" event={"ID":"97dd18ff-36f0-45f4-8b65-0fe85052c2c4","Type":"ContainerStarted","Data":"dc560a34bc693e015f105df051ad32f59ff40e657cbd91578b95422531129b17"}
Jul 28 17:23:25.158748 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.158434    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:25.160388 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.160368    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn"
Jul 28 17:23:25.160632 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.160609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" event={"ID":"30c1a0b6-194c-4bd5-be83-b9ce463e176b","Type":"ContainerStarted","Data":"856b626c28627c26f51faca3327ff73a086c857481c07939c2ca3f72c5f42a10"}
Jul 28 17:23:25.160708 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.160646    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" event={"ID":"30c1a0b6-194c-4bd5-be83-b9ce463e176b","Type":"ContainerStarted","Data":"198a88b46e661e666898f7afb0b546f066291fce5cafa9d70bae7cf29b298eef"}
Jul 28 17:23:25.160708 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.160664    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" event={"ID":"30c1a0b6-194c-4bd5-be83-b9ce463e176b","Type":"ContainerStarted","Data":"3a594aefd38ecb2db50bc12026d5773ee34028ec9d46c49194e42503dd151fa6"}
Jul 28 17:23:25.162058 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.162038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6wjcw" event={"ID":"5edec3e6-cd2b-40b0-a939-e91932b835c8","Type":"ContainerStarted","Data":"dc4b2d533a268ba393250641593c1095c5422b1eb84028b52a4cd9dfddbe86e9"}
Jul 28 17:23:25.165183 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.165162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxvgt" event={"ID":"68af73e0-884e-457e-9e5b-17a3ded1a5c3","Type":"ContainerStarted","Data":"791bcce4218497ab4d9a0208338a61f6f47d46a2af67b672919a73a4637c5319"}
Jul 28 17:23:25.166343 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.166324    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9" event={"ID":"87208fe0-5665-4ff9-8a45-78900fa38288","Type":"ContainerStarted","Data":"18f9de7668731aed3a6bab5558502d1b0c15c63459f8257b444fc9dca23fa047"}
Jul 28 17:23:25.175486 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.175448    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-677d6f476c-s9fmn" podStartSLOduration=14.282151659 podStartE2EDuration="22.175436455s" totalImagesPullingTime="7.893284796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:25.175148169 +0000 UTC m=+45.829643155" watchObservedRunningTime="2026-07-28 17:23:25.175436455 +0000 UTC m=+45.829931420"
Jul 28 17:23:25.193079 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.193029    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78c8bdddc6-dkzp9" podStartSLOduration=14.295970141 podStartE2EDuration="22.19301318s" totalImagesPullingTime="7.897043039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:25.192256592 +0000 UTC m=+45.846751567" watchObservedRunningTime="2026-07-28 17:23:25.19301318 +0000 UTC m=+45.847508169"
Jul 28 17:23:25.208538 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.208491    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6wjcw" podStartSLOduration=3.320491009 podStartE2EDuration="11.208468769s" totalImagesPullingTime="7.88797776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:25.207623745 +0000 UTC m=+45.862118720" watchObservedRunningTime="2026-07-28 17:23:25.208468769 +0000 UTC m=+45.862963785"
Jul 28 17:23:25.230047 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.230006    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" podStartSLOduration=14.316707988 podStartE2EDuration="22.229988436s" totalImagesPullingTime="7.913280448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:25.229793836 +0000 UTC m=+45.884288811" watchObservedRunningTime="2026-07-28 17:23:25.229988436 +0000 UTC m=+45.884483415"
Jul 28 17:23:25.257239 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.257197    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dxvgt" podStartSLOduration=17.16675419 podStartE2EDuration="46.257180932s" totalImagesPullingTime="29.090426742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:25.256069583 +0000 UTC m=+45.910564559" watchObservedRunningTime="2026-07-28 17:23:25.257180932 +0000 UTC m=+45.911675932"
Jul 28 17:23:25.396395 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.396358    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:25.493232 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.493152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:25.801431 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.801350    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:25.899241 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:25.899205    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:27.542953 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:27.542919    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:27.607711 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:27.607654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:30.349527 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:30.349487    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:30.350000 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:30.349643    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:23:30.350000 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:30.349657    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb87c65cd-cqhwp: secret "image-registry-tls" not found
Jul 28 17:23:30.350000 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:30.349722    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls podName:7890b5d6-939f-420e-bfd1-73793793f024 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:46.349704816 +0000 UTC m=+67.004199772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls") pod "image-registry-5bb87c65cd-cqhwp" (UID: "7890b5d6-939f-420e-bfd1-73793793f024") : secret "image-registry-tls" not found
Jul 28 17:23:30.450616 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:30.450582    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:30.450755 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:30.450733    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:30.450809 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:30.450801    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls podName:9410f0ea-7664-4c08-986d-7950cfcabfc7 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:46.450785911 +0000 UTC m=+67.105280864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls") pod "dns-default-496j7" (UID: "9410f0ea-7664-4c08-986d-7950cfcabfc7") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:39.505921 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:39.505866    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:39.604442 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:39.604410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:39.672778 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:39.672746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:40.496918 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:40.496869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:40.581335 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:40.581298    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:41.128412 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:41.128384    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nrwj7"
Jul 28 17:23:41.286091 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:41.286038    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:41.451398 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:41.451317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:44.496072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:44.496036    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:45.563181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.563139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:45.565674 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.565654    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:23:45.577162 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.577135    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c6f3233-64d4-424d-9f24-b583d6beacf7-original-pull-secret\") pod \"global-pull-secret-syncer-rff9b\" (UID: \"1c6f3233-64d4-424d-9f24-b583d6beacf7\") " pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:45.664182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.664154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:23:45.666576 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.666555    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:23:45.674578 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:45.674559    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:23:45.674693 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:45.674657    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs podName:39014b65-63d6-416a-90bd-9baf56efe171 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:49.674631608 +0000 UTC m=+130.329126565 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs") pod "network-metrics-daemon-89z9z" (UID: "39014b65-63d6-416a-90bd-9baf56efe171") : secret "metrics-daemon-secret" not found
Jul 28 17:23:45.764690 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.764650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:45.767150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.767132    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:23:45.776981 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.776965    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:23:45.787997 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.787979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ptk\" (UniqueName: \"kubernetes.io/projected/42477d3f-a75e-48bc-89d5-176f6a162f00-kube-api-access-x7ptk\") pod \"network-check-target-jzkpr\" (UID: \"42477d3f-a75e-48bc-89d5-176f6a162f00\") " pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:45.838026 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.837947    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rff9b"
Jul 28 17:23:45.965053 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:45.965019    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rff9b"]
Jul 28 17:23:45.968741 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:23:45.968718    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6f3233_64d4_424d_9f24_b583d6beacf7.slice/crio-ae124052bd0f6c9eec0c5a5f00c7eeb14b3a13c091f013d9c44950e7d8bad87f WatchSource:0}: Error finding container ae124052bd0f6c9eec0c5a5f00c7eeb14b3a13c091f013d9c44950e7d8bad87f: Status 404 returned error can't find the container with id ae124052bd0f6c9eec0c5a5f00c7eeb14b3a13c091f013d9c44950e7d8bad87f
Jul 28 17:23:46.058483 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:46.058454    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rqpml\""
Jul 28 17:23:46.067161 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:46.067138    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:46.189917 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:46.189861    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jzkpr"]
Jul 28 17:23:46.194695 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:23:46.194671    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42477d3f_a75e_48bc_89d5_176f6a162f00.slice/crio-37413c509feb32b610870acee4526e11014d23bed0766a4d93234e5bc21ec792 WatchSource:0}: Error finding container 37413c509feb32b610870acee4526e11014d23bed0766a4d93234e5bc21ec792: Status 404 returned error can't find the container with id 37413c509feb32b610870acee4526e11014d23bed0766a4d93234e5bc21ec792
Jul 28 17:23:46.235064 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:46.235020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rff9b" event={"ID":"1c6f3233-64d4-424d-9f24-b583d6beacf7","Type":"ContainerStarted","Data":"ae124052bd0f6c9eec0c5a5f00c7eeb14b3a13c091f013d9c44950e7d8bad87f"}
Jul 28 17:23:46.236077 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:46.236057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jzkpr" event={"ID":"42477d3f-a75e-48bc-89d5-176f6a162f00","Type":"ContainerStarted","Data":"37413c509feb32b610870acee4526e11014d23bed0766a4d93234e5bc21ec792"}
Jul 28 17:23:46.370170 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:46.370077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:23:46.370328 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:46.370212    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:23:46.370328 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:46.370229    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb87c65cd-cqhwp: secret "image-registry-tls" not found
Jul 28 17:23:46.370328 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:46.370309    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls podName:7890b5d6-939f-420e-bfd1-73793793f024 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:18.370288777 +0000 UTC m=+99.024783744 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls") pod "image-registry-5bb87c65cd-cqhwp" (UID: "7890b5d6-939f-420e-bfd1-73793793f024") : secret "image-registry-tls" not found
Jul 28 17:23:46.470732 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:46.470694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:23:46.470935 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:46.470835    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:46.470935 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:23:46.470925    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls podName:9410f0ea-7664-4c08-986d-7950cfcabfc7 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:18.470903867 +0000 UTC m=+99.125398821 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls") pod "dns-default-496j7" (UID: "9410f0ea-7664-4c08-986d-7950cfcabfc7") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:48.767644 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:48.767596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:48.833247 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:48.833202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:51.255612 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:51.255575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jzkpr" event={"ID":"42477d3f-a75e-48bc-89d5-176f6a162f00","Type":"ContainerStarted","Data":"a4bffa67a2d9befd639ea11a5dd6de74d3a9fd650d2efad702244b22918fb809"}
Jul 28 17:23:51.256097 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:51.255690    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:23:51.257127 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:51.257104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rff9b" event={"ID":"1c6f3233-64d4-424d-9f24-b583d6beacf7","Type":"ContainerStarted","Data":"1f4e11d807eb98c6d307e0fcdee65202c9ee1326954afdbd80cf283f39126905"}
Jul 28 17:23:51.272143 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:51.272091    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jzkpr" podStartSLOduration=66.702092818 podStartE2EDuration="1m11.272074966s" totalImagesPullingTime="4.569982148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:51.271599317 +0000 UTC m=+71.926094291" watchObservedRunningTime="2026-07-28 17:23:51.272074966 +0000 UTC m=+71.926569931"
Jul 28 17:23:51.285932 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:51.285864    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rff9b" podStartSLOduration=66.485330304 podStartE2EDuration="1m11.285846139s" totalImagesPullingTime="4.800515835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:51.285505609 +0000 UTC m=+71.940000586" watchObservedRunningTime="2026-07-28 17:23:51.285846139 +0000 UTC m=+71.940341115"
Jul 28 17:23:52.604481 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:52.604445    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:52.668537 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:52.668504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:55.625181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:55.625142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:55.732379 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:55.732343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:55.830466 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:55.830433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:55.930855 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:55.930772    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:57.360644 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:57.360607    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:57.462806 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:23:57.462763    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:00.427139 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.427106    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nvtkq"]
Jul 28 17:24:00.429885 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.429850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.432546 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.432526    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:24:00.432995 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.432978    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:24:00.433095 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.433020    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7f8h\""
Jul 28 17:24:00.433497 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.433481    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:24:00.434478 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.434459    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:24:00.449027 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.449007    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nvtkq"]
Jul 28 17:24:00.471489 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.471443    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:00.487395 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.487357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9979e836-b374-4a29-b553-486da46ce962-crio-socket\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.487395 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.487402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.487560 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.487420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7st\" (UniqueName: \"kubernetes.io/projected/9979e836-b374-4a29-b553-486da46ce962-kube-api-access-hg7st\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.487560 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.487487    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.487560 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.487543    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9979e836-b374-4a29-b553-486da46ce962-data-volume\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.588862 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.588826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.588862 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.588863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7st\" (UniqueName: \"kubernetes.io/projected/9979e836-b374-4a29-b553-486da46ce962-kube-api-access-hg7st\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.589064 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:00.588948    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:00.589064 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.588996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.589064 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:00.589031    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:01.089009828 +0000 UTC m=+81.743504802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:00.589193 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.589079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9979e836-b374-4a29-b553-486da46ce962-data-volume\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.589193 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:00.589084    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:00.589193 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.589185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9979e836-b374-4a29-b553-486da46ce962-crio-socket\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.589326 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:00.589257    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:01.089234377 +0000 UTC m=+81.743729347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:00.589326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.589256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9979e836-b374-4a29-b553-486da46ce962-crio-socket\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.589395 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.589374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9979e836-b374-4a29-b553-486da46ce962-data-volume\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.597483 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.597455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7st\" (UniqueName: \"kubernetes.io/projected/9979e836-b374-4a29-b553-486da46ce962-kube-api-access-hg7st\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:00.683053 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:00.682969    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:01.093334 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:01.093238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:01.093334 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:01.093290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:01.093528 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:01.093352    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:01.093528 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:01.093438    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:01.093528 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:01.093444    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.093425845 +0000 UTC m=+82.747920828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:01.093528 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:01.093507    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.093494747 +0000 UTC m=+82.747989700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:02.102005 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:02.101965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:02.102005 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:02.102007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:02.102405 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:02.102099    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:02.102405 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:02.102121    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:02.102405 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:02.102165    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:04.102149786 +0000 UTC m=+84.756644739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:02.102405 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:02.102179    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:04.102173498 +0000 UTC m=+84.756668451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:02.578999 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:02.578967    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mtzfw_98966b68-fcf4-411b-9db9-20fd9012cd26/dns-node-resolver/0.log"
Jul 28 17:24:03.179070 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:03.179041    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-986jm_4382b057-9b5d-43fb-8d3e-a64b33732a92/node-ca/0.log"
Jul 28 17:24:03.778590 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:03.778557    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6wjcw_5edec3e6-cd2b-40b0-a939-e91932b835c8/serve-healthcheck-canary/0.log"
Jul 28 17:24:04.121402 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:04.121371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:04.121561 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:04.121411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:04.121561 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:04.121492    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:04.121561 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:04.121508    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:04.121561 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:04.121558    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:08.121544885 +0000 UTC m=+88.776039838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:04.121700 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:04.121571    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:08.121564938 +0000 UTC m=+88.776059891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:08.155544 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:08.155509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:08.155994 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:08.155551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:08.155994 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:08.155660    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:08.155994 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:08.155657    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:08.155994 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:08.155728    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:16.155713306 +0000 UTC m=+96.810208259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:08.155994 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:08.155742    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:16.155735617 +0000 UTC m=+96.810230571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:08.855152 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:08.855111    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:08.919377 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:08.919341    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:11.113595 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:11.113561    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:11.181314 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:11.181282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:11.361369 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:11.361334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:11.461297 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:11.461212    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:11.561385 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:11.561353    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:11.623007 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:11.622971    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:11.841444 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:11.841362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:11.944636 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:11.944599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:12.409749 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:12.409713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:12.514072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:12.514044    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:13.446925 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:13.446893    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:13.548130 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:13.548100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:13.651245 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:13.651214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:14.466344 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:14.466307    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:16.227070 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:16.227031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:16.227070 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:16.227072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:16.227477 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:16.227225    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:16.227477 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:16.227310    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls podName:9979e836-b374-4a29-b553-486da46ce962 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:32.227290059 +0000 UTC m=+112.881785014 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvtkq" (UID: "9979e836-b374-4a29-b553-486da46ce962") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:16.227640 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:16.227618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9979e836-b374-4a29-b553-486da46ce962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:17.213629 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:17.213591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:17.296619 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:17.296581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:17.724276 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:17.724242    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:17.822701 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:17.822664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:17.923433 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:17.923402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:18.024568 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:18.024460    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:18.124292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:18.124262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:18.225952 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:18.225914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:18.446204 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:18.446169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") pod \"image-registry-5bb87c65cd-cqhwp\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") " pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:24:18.446569 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:18.446314    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:24:18.446569 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:18.446329    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bb87c65cd-cqhwp: secret "image-registry-tls" not found
Jul 28 17:24:18.446569 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:18.446395    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls podName:7890b5d6-939f-420e-bfd1-73793793f024 nodeName:}" failed. No retries permitted until 2026-07-28 17:25:22.446377442 +0000 UTC m=+163.100872400 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls") pod "image-registry-5bb87c65cd-cqhwp" (UID: "7890b5d6-939f-420e-bfd1-73793793f024") : secret "image-registry-tls" not found
Jul 28 17:24:18.547148 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:18.547109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:24:18.547322 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:18.547260    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:24:18.547374 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:18.547344    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls podName:9410f0ea-7664-4c08-986d-7950cfcabfc7 nodeName:}" failed. No retries permitted until 2026-07-28 17:25:22.547323458 +0000 UTC m=+163.201818410 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls") pod "dns-default-496j7" (UID: "9410f0ea-7664-4c08-986d-7950cfcabfc7") : secret "dns-default-metrics-tls" not found
Jul 28 17:24:18.617338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:18.617304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:19.024375 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:19.024340    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:19.621489 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:19.621454    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:20.241639 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:20.241599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:20.438577 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:20.438541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:20.641162 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:20.641130    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:21.026400 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:21.026304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:21.364467 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:21.364438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:21.615318 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:21.615226    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:21.674770 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:21.674737    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:22.021414 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:22.021313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:22.262291 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:22.262264    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jzkpr"
Jul 28 17:24:22.424823 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:22.424789    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:22.627463 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:22.627432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:23.023987 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:23.023951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:23.624781 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:23.624742    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:23.738100 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:23.738066    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:24.019653 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:24.019568    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:24.222782 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:24.222748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:24.629406 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:24.629374    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:25.221985 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:25.221952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:25.618386 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:25.618353    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:25.836290 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:25.836264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:26.229783 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:26.229750    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:26.364130 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:26.364098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:26.618048 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:26.618020    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:27.031605 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:27.031488    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:27.621920 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:27.621888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:28.011416 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:28.011338    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:28.217984 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:28.217943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:28.416799 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:28.416765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:29.466939 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:29.466892    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:29.538717 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:29.538684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:29.733505 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:29.733417    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:29.913371 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:29.913337    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:30.127738 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:30.127705    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:30.188751 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:30.188715    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:30.509104 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:30.509025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:30.931633 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:30.931599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:31.128078 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:31.128043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:31.321335 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:31.321204    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:32.000642 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:32.000604    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:32.122204 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:32.122170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:32.224177 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:32.224122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:32.245661 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:32.245630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:32.248052 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:32.248024    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9979e836-b374-4a29-b553-486da46ce962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvtkq\" (UID: \"9979e836-b374-4a29-b553-486da46ce962\") " pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:32.294015 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:32.293952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:32.538753 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:32.538722    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nvtkq"
Jul 28 17:24:32.666338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:32.666310    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nvtkq"]
Jul 28 17:24:32.669258 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:24:32.669229    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9979e836_b374_4a29_b553_486da46ce962.slice/crio-d90022300343f19351decd7f27dfe4836c93ad4fdf3ef2b0a3fd41762d6bf316 WatchSource:0}: Error finding container d90022300343f19351decd7f27dfe4836c93ad4fdf3ef2b0a3fd41762d6bf316: Status 404 returned error can't find the container with id d90022300343f19351decd7f27dfe4836c93ad4fdf3ef2b0a3fd41762d6bf316
Jul 28 17:24:33.136620 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:33.136591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:33.384895 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:33.384836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nvtkq" event={"ID":"9979e836-b374-4a29-b553-486da46ce962","Type":"ContainerStarted","Data":"d90022300343f19351decd7f27dfe4836c93ad4fdf3ef2b0a3fd41762d6bf316"}
Jul 28 17:24:34.863473 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:34.863424    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" podUID="30c1a0b6-194c-4bd5-be83-b9ce463e176b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:24:36.088842 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:36.088808    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:36.202940 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:36.202898    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:36.298293 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:36.298259    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:38.404360 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:38.404282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nvtkq" event={"ID":"9979e836-b374-4a29-b553-486da46ce962","Type":"ContainerStarted","Data":"477e4548550fd2cbf615d7a45f28dcaae1f83916a90d8e9f5f24be73452ee756"}
Jul 28 17:24:39.747416 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:39.747380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:39.846449 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:39.846415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:40.010569 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:40.010447    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:40.073379 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:40.073306    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:41.186139 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:41.186088    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:41.265063 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:41.265029    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:41.395193 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:41.395158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:41.415748 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:41.415716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nvtkq" event={"ID":"9979e836-b374-4a29-b553-486da46ce962","Type":"ContainerStarted","Data":"ba6643934da9608a9181c3054c7e6a068d2eca0ba3aa010c003b7a5eccf06716"}
Jul 28 17:24:42.143586 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:42.143523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:42.248635 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:42.248595    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:44.298637 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:44.298594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:44.862954 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:44.862921    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" podUID="30c1a0b6-194c-4bd5-be83-b9ce463e176b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:24:45.430721 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:45.430691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nvtkq" event={"ID":"9979e836-b374-4a29-b553-486da46ce962","Type":"ContainerStarted","Data":"25b7ddecd617746183226aec4cc4246645a534ccc8abbdeae48acff976e9781e"}
Jul 28 17:24:45.449684 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:45.449630    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nvtkq" podStartSLOduration=33.434183219 podStartE2EDuration="45.449611124s" totalImagesPullingTime="12.015427905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:24:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:45.448639788 +0000 UTC m=+126.103134763" watchObservedRunningTime="2026-07-28 17:24:45.449611124 +0000 UTC m=+126.104106099"
Jul 28 17:24:45.538007 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:45.537962    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:45.595624 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:45.595585    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:46.328809 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.328777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:46.513557 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.513515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:46.650621 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.650588    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4nmcq"]
Jul 28 17:24:46.653677 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.653660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.656096 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.656073    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:24:46.656250 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.656146    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n84sh\""
Jul 28 17:24:46.656680 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.656662    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:24:46.661914 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.661893    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:24:46.661914 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.661903    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:24:46.661914 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.661895    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:24:46.662100 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.661921    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:24:46.739158 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.739122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:46.766215 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.766187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-wtmp\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.766308 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.766268    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c969b397-8973-4a66-be48-9911f7e203e7-metrics-client-ca\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.766308 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.766299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.766397 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.766320    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c969b397-8973-4a66-be48-9911f7e203e7-sys\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.766397 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.766368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clb42\" (UniqueName: \"kubernetes.io/projected/c969b397-8973-4a66-be48-9911f7e203e7-kube-api-access-clb42\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.766475 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.766415    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c969b397-8973-4a66-be48-9911f7e203e7-root\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.766475 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.766433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.766475 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.766453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-textfile\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.766475 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.766469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-tls\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.827397 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.827362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:46.867211 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867211 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c969b397-8973-4a66-be48-9911f7e203e7-sys\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clb42\" (UniqueName: \"kubernetes.io/projected/c969b397-8973-4a66-be48-9911f7e203e7-kube-api-access-clb42\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c969b397-8973-4a66-be48-9911f7e203e7-root\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867265    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-textfile\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867296    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c969b397-8973-4a66-be48-9911f7e203e7-sys\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c969b397-8973-4a66-be48-9911f7e203e7-root\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867308    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-tls\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:46.867385    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-wtmp\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867441 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:24:46.867438    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-tls podName:c969b397-8973-4a66-be48-9911f7e203e7 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:47.367419216 +0000 UTC m=+128.021914169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-tls") pod "node-exporter-4nmcq" (UID: "c969b397-8973-4a66-be48-9911f7e203e7") : secret "node-exporter-tls" not found
Jul 28 17:24:46.867869 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c969b397-8973-4a66-be48-9911f7e203e7-metrics-client-ca\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867869 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-wtmp\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.867869 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-textfile\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.868006 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.867946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.868066 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.868051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c969b397-8973-4a66-be48-9911f7e203e7-metrics-client-ca\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.869669 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.869652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.876077 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.876048    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clb42\" (UniqueName: \"kubernetes.io/projected/c969b397-8973-4a66-be48-9911f7e203e7-kube-api-access-clb42\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:46.931751 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:46.931666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:47.034375 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.034317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:47.162414 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.162375    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:47.263198 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.263106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:47.371773 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.371733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-tls\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:47.374235 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.374211    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c969b397-8973-4a66-be48-9911f7e203e7-node-exporter-tls\") pod \"node-exporter-4nmcq\" (UID: \"c969b397-8973-4a66-be48-9911f7e203e7\") " pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:47.547263 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.547138    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:47.562099 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.562073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4nmcq"
Jul 28 17:24:47.573312 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:24:47.573284    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc969b397_8973_4a66_be48_9911f7e203e7.slice/crio-16f2c2eb0c0e5da80fe4596e420c41d9887d5022c35f5d9fce1a1f9aa0a1d8b0 WatchSource:0}: Error finding container 16f2c2eb0c0e5da80fe4596e420c41d9887d5022c35f5d9fce1a1f9aa0a1d8b0: Status 404 returned error can't find the container with id 16f2c2eb0c0e5da80fe4596e420c41d9887d5022c35f5d9fce1a1f9aa0a1d8b0
Jul 28 17:24:47.645702 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.645669    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:47.746314 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.746271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:47.830382 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:47.830293    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:48.156978 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:48.156932    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:48.227902 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:48.227839    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:48.441800 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:48.441730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4nmcq" event={"ID":"c969b397-8973-4a66-be48-9911f7e203e7","Type":"ContainerStarted","Data":"16f2c2eb0c0e5da80fe4596e420c41d9887d5022c35f5d9fce1a1f9aa0a1d8b0"}
Jul 28 17:24:48.684958 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:48.684927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:48.980762 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:48.980734    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:49.446817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.446787    2559 generic.go:356] "Generic (PLEG): container finished" podID="c969b397-8973-4a66-be48-9911f7e203e7" containerID="b3edff7414e00e77b7db9eb03cb3f2aa6e9562b9b5c157a654733ef0447289c8" exitCode=0
Jul 28 17:24:49.446984 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.446851    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4nmcq" event={"ID":"c969b397-8973-4a66-be48-9911f7e203e7","Type":"ContainerDied","Data":"b3edff7414e00e77b7db9eb03cb3f2aa6e9562b9b5c157a654733ef0447289c8"}
Jul 28 17:24:49.579316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.579287    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:49.642954 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.642922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:49.692418 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.692387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:24:49.694759 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.694742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39014b65-63d6-416a-90bd-9baf56efe171-metrics-certs\") pod \"network-metrics-daemon-89z9z\" (UID: \"39014b65-63d6-416a-90bd-9baf56efe171\") " pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:24:49.735850 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.735772    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:49.830236 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.830170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:49.912582 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.912550    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:49.953035 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.953010    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dpv4x\""
Jul 28 17:24:49.961330 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:49.961312    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-89z9z"
Jul 28 17:24:50.089439 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:50.089404    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-89z9z"]
Jul 28 17:24:50.092506 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:24:50.092475    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39014b65_63d6_416a_90bd_9baf56efe171.slice/crio-c315efe0e558b85123e13311f23cc446b0d1ba32c7373a667831fe6bdb393771 WatchSource:0}: Error finding container c315efe0e558b85123e13311f23cc446b0d1ba32c7373a667831fe6bdb393771: Status 404 returned error can't find the container with id c315efe0e558b85123e13311f23cc446b0d1ba32c7373a667831fe6bdb393771
Jul 28 17:24:50.281080 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:50.280991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:50.451743 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:50.451709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-89z9z" event={"ID":"39014b65-63d6-416a-90bd-9baf56efe171","Type":"ContainerStarted","Data":"c315efe0e558b85123e13311f23cc446b0d1ba32c7373a667831fe6bdb393771"}
Jul 28 17:24:50.453804 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:50.453781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4nmcq" event={"ID":"c969b397-8973-4a66-be48-9911f7e203e7","Type":"ContainerStarted","Data":"84275b264ae62c2f97eef8c01c89b6de73814a626793681b9f8aec41c439abd3"}
Jul 28 17:24:50.453983 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:50.453818    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4nmcq" event={"ID":"c969b397-8973-4a66-be48-9911f7e203e7","Type":"ContainerStarted","Data":"c6fa8d3b56e16c05588c92db180e93bb5e9d1e69a80cdd18f3a482b9284174ac"}
Jul 28 17:24:50.482077 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:50.482028    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4nmcq" podStartSLOduration=3.509376056 podStartE2EDuration="4.482013233s" totalImagesPullingTime="903.81884ms" totalInitContainerRuntime="68.818337ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:50.481097438 +0000 UTC m=+131.135592406" watchObservedRunningTime="2026-07-28 17:24:50.482013233 +0000 UTC m=+131.136508208"
Jul 28 17:24:51.372297 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:51.372266    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:51.466087 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:51.466057    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:51.595397 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:51.595365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:52.462959 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:52.462924    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-89z9z" event={"ID":"39014b65-63d6-416a-90bd-9baf56efe171","Type":"ContainerStarted","Data":"4d9f2010a3edb904c677c296fc2b6dbbf04a5c0911e3a7589dc52921672f8941"}
Jul 28 17:24:52.463388 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:52.462974    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-89z9z" event={"ID":"39014b65-63d6-416a-90bd-9baf56efe171","Type":"ContainerStarted","Data":"702b294143241c02c17f2dbd0bbf029e3b1326f14c27264b5f93f3a4de0fdef9"}
Jul 28 17:24:52.625797 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:52.625761    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:52.733389 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:52.733301    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:52.838031 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:52.837999    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:52.933374 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:52.933338    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:53.043408 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:53.043325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:54.863074 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:54.863038    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" podUID="30c1a0b6-194c-4bd5-be83-b9ce463e176b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:24:54.863435 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:54.863103    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h"
Jul 28 17:24:54.863565 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:54.863536    2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"856b626c28627c26f51faca3327ff73a086c857481c07939c2ca3f72c5f42a10"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 28 17:24:54.863602 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:54.863584    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" podUID="30c1a0b6-194c-4bd5-be83-b9ce463e176b" containerName="service-proxy" containerID="cri-o://856b626c28627c26f51faca3327ff73a086c857481c07939c2ca3f72c5f42a10" gracePeriod=30
Jul 28 17:24:55.338284 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:55.338195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:55.476252 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:55.476213    2559 generic.go:356] "Generic (PLEG): container finished" podID="30c1a0b6-194c-4bd5-be83-b9ce463e176b" containerID="856b626c28627c26f51faca3327ff73a086c857481c07939c2ca3f72c5f42a10" exitCode=2
Jul 28 17:24:55.476410 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:55.476283    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" event={"ID":"30c1a0b6-194c-4bd5-be83-b9ce463e176b","Type":"ContainerDied","Data":"856b626c28627c26f51faca3327ff73a086c857481c07939c2ca3f72c5f42a10"}
Jul 28 17:24:55.476410 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:55.476341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdcf56496-k5f9h" event={"ID":"30c1a0b6-194c-4bd5-be83-b9ce463e176b","Type":"ContainerStarted","Data":"e2ea0e7cc601900a13dd3d51986aec25925670cc8df86527ca757639eac515f2"}
Jul 28 17:24:55.497656 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:55.497610    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-89z9z" podStartSLOduration=134.144014362 podStartE2EDuration="2m15.497595905s" totalImagesPullingTime="1.353581543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:52.490640781 +0000 UTC m=+133.145135758" watchObservedRunningTime="2026-07-28 17:24:55.497595905 +0000 UTC m=+136.152090888"
Jul 28 17:24:57.196612 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:57.196580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:57.255140 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:57.255109    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:58.572583 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:58.572556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:58.672124 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:24:58.672094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:00.150699 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:00.150659    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:00.252366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:00.252333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:00.626972 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:00.626940    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bb87c65cd-cqhwp"]
Jul 28 17:25:00.627166 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:25:00.627149    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp" podUID="7890b5d6-939f-420e-bfd1-73793793f024"
Jul 28 17:25:00.725137 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:00.725106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:00.830106 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:00.830081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:00.932042 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:00.931954    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:01.497117 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.497089    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:25:01.503539 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.503513    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:25:01.593493 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.593462    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-trusted-ca\") pod \"7890b5d6-939f-420e-bfd1-73793793f024\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") "
Jul 28 17:25:01.593696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.593505    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-registry-certificates\") pod \"7890b5d6-939f-420e-bfd1-73793793f024\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") "
Jul 28 17:25:01.593696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.593530    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwr6r\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-kube-api-access-nwr6r\") pod \"7890b5d6-939f-420e-bfd1-73793793f024\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") "
Jul 28 17:25:01.593696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.593558    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7890b5d6-939f-420e-bfd1-73793793f024-ca-trust-extracted\") pod \"7890b5d6-939f-420e-bfd1-73793793f024\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") "
Jul 28 17:25:01.593696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.593614    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-bound-sa-token\") pod \"7890b5d6-939f-420e-bfd1-73793793f024\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") "
Jul 28 17:25:01.593696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.593662    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-image-registry-private-configuration\") pod \"7890b5d6-939f-420e-bfd1-73793793f024\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") "
Jul 28 17:25:01.593696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.593690    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-installation-pull-secrets\") pod \"7890b5d6-939f-420e-bfd1-73793793f024\" (UID: \"7890b5d6-939f-420e-bfd1-73793793f024\") "
Jul 28 17:25:01.594019 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.593895    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7890b5d6-939f-420e-bfd1-73793793f024" (UID: "7890b5d6-939f-420e-bfd1-73793793f024"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:01.594075 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.594048    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7890b5d6-939f-420e-bfd1-73793793f024" (UID: "7890b5d6-939f-420e-bfd1-73793793f024"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:01.594181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.594154    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7890b5d6-939f-420e-bfd1-73793793f024-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7890b5d6-939f-420e-bfd1-73793793f024" (UID: "7890b5d6-939f-420e-bfd1-73793793f024"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:01.596181 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.596135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7890b5d6-939f-420e-bfd1-73793793f024" (UID: "7890b5d6-939f-420e-bfd1-73793793f024"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:01.596279 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.596241    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7890b5d6-939f-420e-bfd1-73793793f024" (UID: "7890b5d6-939f-420e-bfd1-73793793f024"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:01.596279 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.596254    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7890b5d6-939f-420e-bfd1-73793793f024" (UID: "7890b5d6-939f-420e-bfd1-73793793f024"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:01.596348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.596283    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-kube-api-access-nwr6r" (OuterVolumeSpecName: "kube-api-access-nwr6r") pod "7890b5d6-939f-420e-bfd1-73793793f024" (UID: "7890b5d6-939f-420e-bfd1-73793793f024"). InnerVolumeSpecName "kube-api-access-nwr6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:01.694789 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.694749    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-installation-pull-secrets\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:25:01.694789 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.694785    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-trusted-ca\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:25:01.695027 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.694801    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7890b5d6-939f-420e-bfd1-73793793f024-registry-certificates\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:25:01.695027 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.694811    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwr6r\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-kube-api-access-nwr6r\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:25:01.695027 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.694820    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7890b5d6-939f-420e-bfd1-73793793f024-ca-trust-extracted\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:25:01.695027 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.694828    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-bound-sa-token\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:25:01.695027 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:01.694836    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7890b5d6-939f-420e-bfd1-73793793f024-image-registry-private-configuration\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:25:02.164338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:02.164299    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:02.263394 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:02.263361    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:02.500502 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:02.500420    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb87c65cd-cqhwp"
Jul 28 17:25:02.535720 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:02.535690    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bb87c65cd-cqhwp"]
Jul 28 17:25:02.543137 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:02.543112    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bb87c65cd-cqhwp"]
Jul 28 17:25:02.702510 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:02.702471    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7890b5d6-939f-420e-bfd1-73793793f024-registry-tls\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:25:03.831751 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:03.831720    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7890b5d6-939f-420e-bfd1-73793793f024" path="/var/lib/kubelet/pods/7890b5d6-939f-420e-bfd1-73793793f024/volumes"
Jul 28 17:25:04.351171 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:04.351135    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:04.414426 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:04.414394    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:09.882007 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:09.881973    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:09.982962 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:09.982929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:10.043007 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:10.042976    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:13.006986 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:13.006957    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:13.287658 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:13.287574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:13.389762 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:13.389726    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:17.593689 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:25:17.593644    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-496j7" podUID="9410f0ea-7664-4c08-986d-7950cfcabfc7"
Jul 28 17:25:18.551614 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:18.551583    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-496j7"
Jul 28 17:25:22.552226 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:22.552188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:25:22.554539 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:22.554520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9410f0ea-7664-4c08-986d-7950cfcabfc7-metrics-tls\") pod \"dns-default-496j7\" (UID: \"9410f0ea-7664-4c08-986d-7950cfcabfc7\") " pod="openshift-dns/dns-default-496j7"
Jul 28 17:25:22.755336 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:22.755303    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gw5z2\""
Jul 28 17:25:22.763118 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:22.763096    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-496j7"
Jul 28 17:25:22.941369 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:22.941327    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-496j7"]
Jul 28 17:25:22.943865 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:25:22.943832    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9410f0ea_7664_4c08_986d_7950cfcabfc7.slice/crio-f585803b233b3e1756d9bdff4b4711789fe0057473a13222fb223c6d0199cda2 WatchSource:0}: Error finding container f585803b233b3e1756d9bdff4b4711789fe0057473a13222fb223c6d0199cda2: Status 404 returned error can't find the container with id f585803b233b3e1756d9bdff4b4711789fe0057473a13222fb223c6d0199cda2
Jul 28 17:25:23.567471 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:23.567428    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-496j7" event={"ID":"9410f0ea-7664-4c08-986d-7950cfcabfc7","Type":"ContainerStarted","Data":"f585803b233b3e1756d9bdff4b4711789fe0057473a13222fb223c6d0199cda2"}
Jul 28 17:25:25.575953 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:25.575919    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-496j7" event={"ID":"9410f0ea-7664-4c08-986d-7950cfcabfc7","Type":"ContainerStarted","Data":"b5cf965b791e256264e5358e69c7f706e73d6075644e43195a926bcecd4cd94b"}
Jul 28 17:25:25.576358 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:25.575973    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-496j7" event={"ID":"9410f0ea-7664-4c08-986d-7950cfcabfc7","Type":"ContainerStarted","Data":"e16808ea0d1a9360abe5e699555e723da584a0f59f13bc4d04016cc189660878"}
Jul 28 17:25:25.576358 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:25.576012    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-496j7"
Jul 28 17:25:25.597323 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:25.597271    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-496j7" podStartSLOduration=129.824953157 podStartE2EDuration="2m11.597257016s" totalImagesPullingTime="1.772303859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:25.596436235 +0000 UTC m=+166.250931210" watchObservedRunningTime="2026-07-28 17:25:25.597257016 +0000 UTC m=+166.251752010"
Jul 28 17:25:27.934473 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:27.934427    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:27.991979 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:27.991946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:32.169475 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:32.169440    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:32.268865 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:32.268817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:35.582336 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:35.582307    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-496j7"
Jul 28 17:25:44.750583 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:44.750540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:44.846045 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:44.846005    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:54.087123 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:25:54.087090    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:02.849753 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:02.849716    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:02.962832 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:02.962798    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:11.223499 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:11.223454    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:11.292552 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:11.292515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:11.362053 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:11.362014    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:11.438306 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:11.438271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:11.978348 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:11.978315    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:14.718189 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:14.718156    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:14.817098 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:14.817062    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:19.505815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:19.505776    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:19.599724 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:19.599664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:19.663615 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:19.663587    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:22.218965 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:22.218934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:22.320887 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:22.320851    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:22.391804 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:22.391773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:51.075027 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:51.075000    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:58.239428 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:58.239396    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:58.303100 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:26:58.303069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:28.677356 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:27:28.677319    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:28.777340 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:27:28.777304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:39.765569 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:27:39.765544    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:27:52.285228 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:27:52.285191    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:52.475535 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:27:52.475500    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:58.675538 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:27:58.675501    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:59.101591 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:27:59.101514    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:59.408937 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:27:59.408900    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:00.023071 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.023036    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:00.045966 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.045937    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"]
Jul 28 17:28:00.048965 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.048948    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.054430 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.054404    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 17:28:00.054565 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.054505    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5lwzx\""
Jul 28 17:28:00.054565 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.054545    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 17:28:00.054687 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.054647    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 17:28:00.054774 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.054752    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 17:28:00.054919 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.054849    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 17:28:00.068528 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.068501    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"]
Jul 28 17:28:00.115159 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.115132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:00.123116 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.123093    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdxp\" (UniqueName: \"kubernetes.io/projected/17c17592-b721-4d4d-936b-a6f75b72dfd6-kube-api-access-xjdxp\") pod \"keda-metrics-apiserver-55557df8c-8wdmv\" (UID: \"17c17592-b721-4d4d-936b-a6f75b72dfd6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.123185 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.123122    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/17c17592-b721-4d4d-936b-a6f75b72dfd6-certificates\") pod \"keda-metrics-apiserver-55557df8c-8wdmv\" (UID: \"17c17592-b721-4d4d-936b-a6f75b72dfd6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.123185 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.123159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/17c17592-b721-4d4d-936b-a6f75b72dfd6-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-8wdmv\" (UID: \"17c17592-b721-4d4d-936b-a6f75b72dfd6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.224364 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.224332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/17c17592-b721-4d4d-936b-a6f75b72dfd6-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-8wdmv\" (UID: \"17c17592-b721-4d4d-936b-a6f75b72dfd6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.224544 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.224388    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdxp\" (UniqueName: \"kubernetes.io/projected/17c17592-b721-4d4d-936b-a6f75b72dfd6-kube-api-access-xjdxp\") pod \"keda-metrics-apiserver-55557df8c-8wdmv\" (UID: \"17c17592-b721-4d4d-936b-a6f75b72dfd6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.224544 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.224429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/17c17592-b721-4d4d-936b-a6f75b72dfd6-certificates\") pod \"keda-metrics-apiserver-55557df8c-8wdmv\" (UID: \"17c17592-b721-4d4d-936b-a6f75b72dfd6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.224753 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.224731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/17c17592-b721-4d4d-936b-a6f75b72dfd6-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-8wdmv\" (UID: \"17c17592-b721-4d4d-936b-a6f75b72dfd6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.226985 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.226964    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/17c17592-b721-4d4d-936b-a6f75b72dfd6-certificates\") pod \"keda-metrics-apiserver-55557df8c-8wdmv\" (UID: \"17c17592-b721-4d4d-936b-a6f75b72dfd6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.239620 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.239589    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdxp\" (UniqueName: \"kubernetes.io/projected/17c17592-b721-4d4d-936b-a6f75b72dfd6-kube-api-access-xjdxp\") pod \"keda-metrics-apiserver-55557df8c-8wdmv\" (UID: \"17c17592-b721-4d4d-936b-a6f75b72dfd6\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.288164 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.288077    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:00.334033 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.334006    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-j42jn"]
Jul 28 17:28:00.336804 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.336789    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:00.340847 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.340830    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 17:28:00.359626 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.359603    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:00.365255 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.365229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-j42jn"]
Jul 28 17:28:00.426099 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.426070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c28745ab-2384-4e19-a4a4-0bc04d5f369c-certificates\") pod \"keda-admission-56f9798bb4-j42jn\" (UID: \"c28745ab-2384-4e19-a4a4-0bc04d5f369c\") " pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:00.426238 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.426119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkzh\" (UniqueName: \"kubernetes.io/projected/c28745ab-2384-4e19-a4a4-0bc04d5f369c-kube-api-access-zxkzh\") pod \"keda-admission-56f9798bb4-j42jn\" (UID: \"c28745ab-2384-4e19-a4a4-0bc04d5f369c\") " pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:00.501700 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.501671    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"]
Jul 28 17:28:00.505086 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:28:00.505056    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c17592_b721_4d4d_936b_a6f75b72dfd6.slice/crio-b950cf150a8722de0083d3cdbeee0cdcca6d246ddcc3b1864558e204eb9d0762 WatchSource:0}: Error finding container b950cf150a8722de0083d3cdbeee0cdcca6d246ddcc3b1864558e204eb9d0762: Status 404 returned error can't find the container with id b950cf150a8722de0083d3cdbeee0cdcca6d246ddcc3b1864558e204eb9d0762
Jul 28 17:28:00.506256 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.506235    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:28:00.526745 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.526717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c28745ab-2384-4e19-a4a4-0bc04d5f369c-certificates\") pod \"keda-admission-56f9798bb4-j42jn\" (UID: \"c28745ab-2384-4e19-a4a4-0bc04d5f369c\") " pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:00.526851 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.526759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkzh\" (UniqueName: \"kubernetes.io/projected/c28745ab-2384-4e19-a4a4-0bc04d5f369c-kube-api-access-zxkzh\") pod \"keda-admission-56f9798bb4-j42jn\" (UID: \"c28745ab-2384-4e19-a4a4-0bc04d5f369c\") " pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:00.529179 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.529159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c28745ab-2384-4e19-a4a4-0bc04d5f369c-certificates\") pod \"keda-admission-56f9798bb4-j42jn\" (UID: \"c28745ab-2384-4e19-a4a4-0bc04d5f369c\") " pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:00.549671 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.549613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkzh\" (UniqueName: \"kubernetes.io/projected/c28745ab-2384-4e19-a4a4-0bc04d5f369c-kube-api-access-zxkzh\") pod \"keda-admission-56f9798bb4-j42jn\" (UID: \"c28745ab-2384-4e19-a4a4-0bc04d5f369c\") " pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:00.576804 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.576777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:00.647077 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.647043    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:00.694194 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.694162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:00.792405 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.792378    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-j42jn"]
Jul 28 17:28:00.795911 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:28:00.795867    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28745ab_2384_4e19_a4a4_0bc04d5f369c.slice/crio-09b2b89035b8213cfb55b889d0d6474144d25229ede724dc86db125edbb9dd8a WatchSource:0}: Error finding container 09b2b89035b8213cfb55b889d0d6474144d25229ede724dc86db125edbb9dd8a: Status 404 returned error can't find the container with id 09b2b89035b8213cfb55b889d0d6474144d25229ede724dc86db125edbb9dd8a
Jul 28 17:28:00.812202 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:00.812145    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:01.064087 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:01.063983    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-j42jn" event={"ID":"c28745ab-2384-4e19-a4a4-0bc04d5f369c","Type":"ContainerStarted","Data":"09b2b89035b8213cfb55b889d0d6474144d25229ede724dc86db125edbb9dd8a"}
Jul 28 17:28:01.065081 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:01.065057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv" event={"ID":"17c17592-b721-4d4d-936b-a6f75b72dfd6","Type":"ContainerStarted","Data":"b950cf150a8722de0083d3cdbeee0cdcca6d246ddcc3b1864558e204eb9d0762"}
Jul 28 17:28:01.116343 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:01.116309    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:01.225253 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:01.225196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:01.319373 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:01.319285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:05.083642 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:05.083601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-j42jn" event={"ID":"c28745ab-2384-4e19-a4a4-0bc04d5f369c","Type":"ContainerStarted","Data":"cace64438b591e03b5b31325ba51948a000ada41a00a7f1a019eb0e92d6238a9"}
Jul 28 17:28:05.084125 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:05.083671    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:05.084990 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:05.084964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv" event={"ID":"17c17592-b721-4d4d-936b-a6f75b72dfd6","Type":"ContainerStarted","Data":"e2d2faa8cc46d79ff74bdd37a38b52d3bd8bcca89081c1841aaca7ba06535f7e"}
Jul 28 17:28:05.085120 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:05.085103    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:05.111186 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:05.111140    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-j42jn" podStartSLOduration=1.821067369 podStartE2EDuration="5.111125673s" totalImagesPullingTime="3.290058304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:28:05.109692575 +0000 UTC m=+325.764187551" watchObservedRunningTime="2026-07-28 17:28:05.111125673 +0000 UTC m=+325.765620648"
Jul 28 17:28:05.137792 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:05.137742    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv" podStartSLOduration=1.555454187 podStartE2EDuration="5.137729805s" totalImagesPullingTime="3.582275618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:28:05.136586345 +0000 UTC m=+325.791081320" watchObservedRunningTime="2026-07-28 17:28:05.137729805 +0000 UTC m=+325.792224779"
Jul 28 17:28:16.092138 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:16.092107    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8wdmv"
Jul 28 17:28:26.090716 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:26.090689    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-j42jn"
Jul 28 17:28:32.568776 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:32.568740    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:33.934743 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:33.934716    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:45.976436 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:28:45.976390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:20.416905 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:20.416860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:20.481581 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:20.481547    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:24.770948 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.770914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:24.863223 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.863192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:24.914132 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.914095    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-db8st"]
Jul 28 17:29:24.917165 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.917144    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:29:24.919607 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.919587    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:29:24.919784 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.919763    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-b84k6\""
Jul 28 17:29:24.920338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.920318    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:29:24.920338 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.920331    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:29:24.928404 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.928381    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-db8st"]
Jul 28 17:29:24.947903 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:24.947866    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:25.026222 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.026142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:25.027101 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.027084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbndr\" (UniqueName: \"kubernetes.io/projected/f4c353be-e8d5-4191-9193-21c36a64f7ae-kube-api-access-gbndr\") pod \"seaweedfs-64568bcd49-db8st\" (UID: \"f4c353be-e8d5-4191-9193-21c36a64f7ae\") " pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:29:25.027155 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.027117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f4c353be-e8d5-4191-9193-21c36a64f7ae-data\") pod \"seaweedfs-64568bcd49-db8st\" (UID: \"f4c353be-e8d5-4191-9193-21c36a64f7ae\") " pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:29:25.127559 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.127529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbndr\" (UniqueName: \"kubernetes.io/projected/f4c353be-e8d5-4191-9193-21c36a64f7ae-kube-api-access-gbndr\") pod \"seaweedfs-64568bcd49-db8st\" (UID: \"f4c353be-e8d5-4191-9193-21c36a64f7ae\") " pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:29:25.127723 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.127575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f4c353be-e8d5-4191-9193-21c36a64f7ae-data\") pod \"seaweedfs-64568bcd49-db8st\" (UID: \"f4c353be-e8d5-4191-9193-21c36a64f7ae\") " pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:29:25.127982 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.127964    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f4c353be-e8d5-4191-9193-21c36a64f7ae-data\") pod \"seaweedfs-64568bcd49-db8st\" (UID: \"f4c353be-e8d5-4191-9193-21c36a64f7ae\") " pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:29:25.128292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.128265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:25.141226 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.141201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbndr\" (UniqueName: \"kubernetes.io/projected/f4c353be-e8d5-4191-9193-21c36a64f7ae-kube-api-access-gbndr\") pod \"seaweedfs-64568bcd49-db8st\" (UID: \"f4c353be-e8d5-4191-9193-21c36a64f7ae\") " pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:29:25.226412 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.226379    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:29:25.355945 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.355912    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-db8st"]
Jul 28 17:29:25.358626 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:29:25.358600    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c353be_e8d5_4191_9193_21c36a64f7ae.slice/crio-feef7c6cd6a77c3cdd871759764dc3a798758f8962f1498cd5354bc4c8dfd21f WatchSource:0}: Error finding container feef7c6cd6a77c3cdd871759764dc3a798758f8962f1498cd5354bc4c8dfd21f: Status 404 returned error can't find the container with id feef7c6cd6a77c3cdd871759764dc3a798758f8962f1498cd5354bc4c8dfd21f
Jul 28 17:29:25.504236 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.504196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:25.606694 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:25.606607    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:29:26.350805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:26.350767    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-db8st" event={"ID":"f4c353be-e8d5-4191-9193-21c36a64f7ae","Type":"ContainerStarted","Data":"feef7c6cd6a77c3cdd871759764dc3a798758f8962f1498cd5354bc4c8dfd21f"}
Jul 28 17:29:28.359850 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:28.359821    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-db8st" event={"ID":"f4c353be-e8d5-4191-9193-21c36a64f7ae","Type":"ContainerStarted","Data":"6ec668e45528835555a81f9d32ebaed0a85377b1e4c81a344d7734099c2f9d1f"}
Jul 28 17:29:28.360223 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:28.359911    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:29:28.376688 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:28.376603    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-db8st" podStartSLOduration=1.615048393 podStartE2EDuration="4.376587443s" totalImagesPullingTime="2.76153905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:28.37567083 +0000 UTC m=+409.030165820" watchObservedRunningTime="2026-07-28 17:29:28.376587443 +0000 UTC m=+409.031082417"
Jul 28 17:29:34.365359 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:29:34.365322    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-db8st"
Jul 28 17:30:09.124616 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:09.124495    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:09.232918 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:09.232857    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:25.363608 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:25.363566    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:26.271132 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:26.271094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:27.568072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.568043    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"]
Jul 28 17:30:27.571081 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.571067    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"
Jul 28 17:30:27.573431 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.573403    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:30:27.573815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.573797    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-cb92n\""
Jul 28 17:30:27.574072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.574056    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 28 17:30:27.588789 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.588770    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"]
Jul 28 17:30:27.660454 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.660400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32552c40-9ab8-4b45-b1c7-5da921653f18-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jnhxd\" (UID: \"32552c40-9ab8-4b45-b1c7-5da921653f18\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"
Jul 28 17:30:27.660624 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.660482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pswn8\" (UniqueName: \"kubernetes.io/projected/32552c40-9ab8-4b45-b1c7-5da921653f18-kube-api-access-pswn8\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jnhxd\" (UID: \"32552c40-9ab8-4b45-b1c7-5da921653f18\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"
Jul 28 17:30:27.760992 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.760957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32552c40-9ab8-4b45-b1c7-5da921653f18-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jnhxd\" (UID: \"32552c40-9ab8-4b45-b1c7-5da921653f18\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"
Jul 28 17:30:27.761117 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.761018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pswn8\" (UniqueName: \"kubernetes.io/projected/32552c40-9ab8-4b45-b1c7-5da921653f18-kube-api-access-pswn8\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jnhxd\" (UID: \"32552c40-9ab8-4b45-b1c7-5da921653f18\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"
Jul 28 17:30:27.761417 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.761399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32552c40-9ab8-4b45-b1c7-5da921653f18-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jnhxd\" (UID: \"32552c40-9ab8-4b45-b1c7-5da921653f18\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"
Jul 28 17:30:27.787501 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.787477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pswn8\" (UniqueName: \"kubernetes.io/projected/32552c40-9ab8-4b45-b1c7-5da921653f18-kube-api-access-pswn8\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jnhxd\" (UID: \"32552c40-9ab8-4b45-b1c7-5da921653f18\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"
Jul 28 17:30:27.879482 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:27.879454    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"
Jul 28 17:30:28.038799 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:28.038764    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd"]
Jul 28 17:30:28.041917 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:30:28.041844    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32552c40_9ab8_4b45_b1c7_5da921653f18.slice/crio-4584e6729ae99d577592782d68794c9fdf566d916875f48b220851116667df5f WatchSource:0}: Error finding container 4584e6729ae99d577592782d68794c9fdf566d916875f48b220851116667df5f: Status 404 returned error can't find the container with id 4584e6729ae99d577592782d68794c9fdf566d916875f48b220851116667df5f
Jul 28 17:30:28.558819 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:28.558783    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd" event={"ID":"32552c40-9ab8-4b45-b1c7-5da921653f18","Type":"ContainerStarted","Data":"4584e6729ae99d577592782d68794c9fdf566d916875f48b220851116667df5f"}
Jul 28 17:30:29.171628 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:29.171539    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:29.274245 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:29.274160    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:31.574521 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:31.574481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd" event={"ID":"32552c40-9ab8-4b45-b1c7-5da921653f18","Type":"ContainerStarted","Data":"6230174e2866103cd781cc9f2bd538d2fab743fed3ecc884aa95b7d3d5748a3e"}
Jul 28 17:30:31.599004 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:31.598956    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jnhxd" podStartSLOduration=1.8775786060000001 podStartE2EDuration="4.598937695s" totalImagesPullingTime="2.721359089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:31.598109147 +0000 UTC m=+472.252604133" watchObservedRunningTime="2026-07-28 17:30:31.598937695 +0000 UTC m=+472.253432669"
Jul 28 17:30:31.746566 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:31.746530    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:31.845639 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:31.845557    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:34.130072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:34.130032    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:34.230245 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:34.230213    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:34.354746 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:34.354713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:35.747529 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:35.747495    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:35.847205 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:35.847162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:35.947713 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:35.947667    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:41.332366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:41.332324    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:44.117939 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.117911    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qf8lw"]
Jul 28 17:30:44.121038 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.121022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:30:44.123857 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.123835    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 17:30:44.124039 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.124016    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hqb68\""
Jul 28 17:30:44.124607 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.124590    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 17:30:44.139231 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.139210    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qf8lw"]
Jul 28 17:30:44.191166 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.191134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f48c3e7-71ff-49c9-89bb-49020350b340-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qf8lw\" (UID: \"2f48c3e7-71ff-49c9-89bb-49020350b340\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:30:44.191166 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.191165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4cwd\" (UniqueName: \"kubernetes.io/projected/2f48c3e7-71ff-49c9-89bb-49020350b340-kube-api-access-s4cwd\") pod \"cert-manager-webhook-6486ffd997-qf8lw\" (UID: \"2f48c3e7-71ff-49c9-89bb-49020350b340\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:30:44.291623 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.291587    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f48c3e7-71ff-49c9-89bb-49020350b340-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qf8lw\" (UID: \"2f48c3e7-71ff-49c9-89bb-49020350b340\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:30:44.291623 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.291624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4cwd\" (UniqueName: \"kubernetes.io/projected/2f48c3e7-71ff-49c9-89bb-49020350b340-kube-api-access-s4cwd\") pod \"cert-manager-webhook-6486ffd997-qf8lw\" (UID: \"2f48c3e7-71ff-49c9-89bb-49020350b340\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:30:44.300739 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.300711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f48c3e7-71ff-49c9-89bb-49020350b340-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qf8lw\" (UID: \"2f48c3e7-71ff-49c9-89bb-49020350b340\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:30:44.300899 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.300801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4cwd\" (UniqueName: \"kubernetes.io/projected/2f48c3e7-71ff-49c9-89bb-49020350b340-kube-api-access-s4cwd\") pod \"cert-manager-webhook-6486ffd997-qf8lw\" (UID: \"2f48c3e7-71ff-49c9-89bb-49020350b340\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:30:44.430007 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.429912    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:30:44.556353 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.556321    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qf8lw"]
Jul 28 17:30:44.559205 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:30:44.559178    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f48c3e7_71ff_49c9_89bb_49020350b340.slice/crio-239e94b1f460d1c9374a0b5ae5290351fc4cc65c0f3c908e8373edbbc2fc7f98 WatchSource:0}: Error finding container 239e94b1f460d1c9374a0b5ae5290351fc4cc65c0f3c908e8373edbbc2fc7f98: Status 404 returned error can't find the container with id 239e94b1f460d1c9374a0b5ae5290351fc4cc65c0f3c908e8373edbbc2fc7f98
Jul 28 17:30:44.622770 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:44.622730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw" event={"ID":"2f48c3e7-71ff-49c9-89bb-49020350b340","Type":"ContainerStarted","Data":"239e94b1f460d1c9374a0b5ae5290351fc4cc65c0f3c908e8373edbbc2fc7f98"}
Jul 28 17:30:47.635703 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:47.635668    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw" event={"ID":"2f48c3e7-71ff-49c9-89bb-49020350b340","Type":"ContainerStarted","Data":"04c4d48e65ec2a890595e70540a636e29a5480d4f60a6105d88fd1708afe0563"}
Jul 28 17:30:47.636127 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:47.635738    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:30:47.652960 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:47.652850    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw" podStartSLOduration=0.708523499 podStartE2EDuration="3.652832638s" totalImagesPullingTime="2.944309139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:47.652545781 +0000 UTC m=+488.307040740" watchObservedRunningTime="2026-07-28 17:30:47.652832638 +0000 UTC m=+488.307327614"
Jul 28 17:30:48.322291 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:48.322260    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:48.375258 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:48.375222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:50.117279 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:50.117242    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:53.641935 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:30:53.641899    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qf8lw"
Jul 28 17:31:08.175298 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:08.175260    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:08.275528 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:08.275490    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:19.889803 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:19.889761    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:24.211195 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:24.211150    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:24.317123 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:24.317093    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:25.518299 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.518271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:25.554815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.554782    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"]
Jul 28 17:31:25.560141 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.560124    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.569034 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.569014    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 17:31:25.569034 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.569027    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 17:31:25.569227 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.569087    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 17:31:25.569227 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.569099    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 17:31:25.570693 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.570677    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rzft8\""
Jul 28 17:31:25.570819 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.570805    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 17:31:25.572512 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.572500    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 17:31:25.581737 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.581719    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"]
Jul 28 17:31:25.595351 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.595332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.595437 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.595368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54prr\" (UniqueName: \"kubernetes.io/projected/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-kube-api-access-54prr\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.595437 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.595398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.595508 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.595443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.595508 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.595472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.595508 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.595499    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.595595 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.595534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.624317 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.624287    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:25.696590 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.696568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.696725 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.696597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.696725 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.696616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.696725 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.696638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.696725 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.696669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.696725 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.696718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.696998 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.696757    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54prr\" (UniqueName: \"kubernetes.io/projected/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-kube-api-access-54prr\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.697340 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.697315    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.699112 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.699090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.699190 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.699164    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.699572 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.699550    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.699654 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.699632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.707520 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.707493    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54prr\" (UniqueName: \"kubernetes.io/projected/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-kube-api-access-54prr\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.707951 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.707893    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2854c-ed88-4cd3-bb49-e4436c4f02ff-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-82lr6\" (UID: \"adb2854c-ed88-4cd3-bb49-e4436c4f02ff\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:25.868850 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:25.868819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:26.023320 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:26.023287    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"]
Jul 28 17:31:26.026178 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:31:26.026152    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb2854c_ed88_4cd3_bb49_e4436c4f02ff.slice/crio-5078b4b9600faca282b001ac8d764d7702b5b80ab682b01c5f9213590c0838fa WatchSource:0}: Error finding container 5078b4b9600faca282b001ac8d764d7702b5b80ab682b01c5f9213590c0838fa: Status 404 returned error can't find the container with id 5078b4b9600faca282b001ac8d764d7702b5b80ab682b01c5f9213590c0838fa
Jul 28 17:31:26.778095 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:26.778047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6" event={"ID":"adb2854c-ed88-4cd3-bb49-e4436c4f02ff","Type":"ContainerStarted","Data":"5078b4b9600faca282b001ac8d764d7702b5b80ab682b01c5f9213590c0838fa"}
Jul 28 17:31:26.884219 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:26.884176    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:28.420662 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:28.420626    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:31:28.420950 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:28.420694    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:31:28.787529 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:28.787444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6" event={"ID":"adb2854c-ed88-4cd3-bb49-e4436c4f02ff","Type":"ContainerStarted","Data":"895d8d3385b1dac7550cea579d5785393bb16585dc946a20363b51e6750b985f"}
Jul 28 17:31:28.787673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:28.787573    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:28.808468 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:28.808416    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6" podStartSLOduration=1.416038597 podStartE2EDuration="3.80840029s" totalImagesPullingTime="2.392361693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:28.807406321 +0000 UTC m=+529.461901298" watchObservedRunningTime="2026-07-28 17:31:28.80840029 +0000 UTC m=+529.462895264"
Jul 28 17:31:29.777035 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:29.776997    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:29.793367 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:29.793316    2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-82lr6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 17:31:29.793513 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:29.793420    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6" podUID="adb2854c-ed88-4cd3-bb49-e4436c4f02ff" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:31:29.874441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:29.874405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:32.793017 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:32.792990    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-82lr6"
Jul 28 17:31:33.299696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.299657    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"]
Jul 28 17:31:33.302861 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.302837    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.306106 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.306086    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-bkm6s\""
Jul 28 17:31:33.316191 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.316165    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"]
Jul 28 17:31:33.345395 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.345356    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:33.356410 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.356380    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.356580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.356417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfscf\" (UniqueName: \"kubernetes.io/projected/67fbff38-b30c-4641-95cc-f266c7219170-kube-api-access-wfscf\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.356580 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.356441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67fbff38-b30c-4641-95cc-f266c7219170-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.356663 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.356575    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.356663 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.356626    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67fbff38-b30c-4641-95cc-f266c7219170-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.356663 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.356649    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.356774 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.356663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.356774 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.356731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.356774 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.356765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67fbff38-b30c-4641-95cc-f266c7219170-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.429111 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.429075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:33.457265 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457234    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67fbff38-b30c-4641-95cc-f266c7219170-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457265 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457268    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457450 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457284    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457450 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457450 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67fbff38-b30c-4641-95cc-f266c7219170-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457450 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457450 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfscf\" (UniqueName: \"kubernetes.io/projected/67fbff38-b30c-4641-95cc-f266c7219170-kube-api-access-wfscf\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457450 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67fbff38-b30c-4641-95cc-f266c7219170-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457724 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457484    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457774 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457724    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457774 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457748    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457864 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457833    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.457945 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.457914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.458152 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.458133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67fbff38-b30c-4641-95cc-f266c7219170-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.459789 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.459768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67fbff38-b30c-4641-95cc-f266c7219170-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.460094 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.460080    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67fbff38-b30c-4641-95cc-f266c7219170-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.468098 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.468073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfscf\" (UniqueName: \"kubernetes.io/projected/67fbff38-b30c-4641-95cc-f266c7219170-kube-api-access-wfscf\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.468632 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.468613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67fbff38-b30c-4641-95cc-f266c7219170-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-pfnkg\" (UID: \"67fbff38-b30c-4641-95cc-f266c7219170\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.614561 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.614533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:33.750660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.750626    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"]
Jul 28 17:31:33.754239 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:31:33.754207    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fbff38_b30c_4641_95cc_f266c7219170.slice/crio-ec20e610ed505ee0d226623157899cf39cd2607d24b557dc13457174a265c241 WatchSource:0}: Error finding container ec20e610ed505ee0d226623157899cf39cd2607d24b557dc13457174a265c241: Status 404 returned error can't find the container with id ec20e610ed505ee0d226623157899cf39cd2607d24b557dc13457174a265c241
Jul 28 17:31:33.810960 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:33.810920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" event={"ID":"67fbff38-b30c-4641-95cc-f266c7219170","Type":"ContainerStarted","Data":"ec20e610ed505ee0d226623157899cf39cd2607d24b557dc13457174a265c241"}
Jul 28 17:31:36.055267 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:36.055227    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:31:36.055565 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:36.055297    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:31:36.055565 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:36.055327    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:31:36.826379 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:36.826338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" event={"ID":"67fbff38-b30c-4641-95cc-f266c7219170","Type":"ContainerStarted","Data":"a1498e0fc22636abe80ef24ac58ebeb1137063bb6532069f34dbc84e07da5ae5"}
Jul 28 17:31:36.847605 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:36.847540    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" podStartSLOduration=1.548666551 podStartE2EDuration="3.84752409s" totalImagesPullingTime="2.298857539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:36.847148826 +0000 UTC m=+537.501643801" watchObservedRunningTime="2026-07-28 17:31:36.84752409 +0000 UTC m=+537.502019065"
Jul 28 17:31:37.067615 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:37.067585    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:37.615528 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:37.615486    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:37.620321 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:37.620299    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:37.832396 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:37.832369    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:37.832566 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:37.832435    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:31:38.898488 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:38.898448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:39.001495 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:39.001470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:42.901451 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:42.901414    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:51.403264 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:51.403223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:52.241908 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:52.241834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:53.025400 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:53.025361    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:55.448086 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:55.448050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:55.547079 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:55.547035    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:56.414192 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:56.414155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:57.744476 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:57.744441    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"]
Jul 28 17:31:57.837571 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:57.837537    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"
Jul 28 17:31:57.841751 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:57.841730    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 17:31:57.842199 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:57.842178    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hbr4h\""
Jul 28 17:31:57.842504 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:57.842484    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 17:31:57.843659 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:57.843643    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"]
Jul 28 17:31:57.959483 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:57.959453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdz2w\" (UniqueName: \"kubernetes.io/projected/60c6f8b3-8313-4b72-a03d-0399f41c5aa8-kube-api-access-zdz2w\") pod \"limitador-operator-controller-manager-774d94476f-dzzbw\" (UID: \"60c6f8b3-8313-4b72-a03d-0399f41c5aa8\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"
Jul 28 17:31:58.023036 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:58.022944    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:58.060068 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:58.060032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdz2w\" (UniqueName: \"kubernetes.io/projected/60c6f8b3-8313-4b72-a03d-0399f41c5aa8-kube-api-access-zdz2w\") pod \"limitador-operator-controller-manager-774d94476f-dzzbw\" (UID: \"60c6f8b3-8313-4b72-a03d-0399f41c5aa8\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"
Jul 28 17:31:58.073255 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:58.073234    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdz2w\" (UniqueName: \"kubernetes.io/projected/60c6f8b3-8313-4b72-a03d-0399f41c5aa8-kube-api-access-zdz2w\") pod \"limitador-operator-controller-manager-774d94476f-dzzbw\" (UID: \"60c6f8b3-8313-4b72-a03d-0399f41c5aa8\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"
Jul 28 17:31:58.148233 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:58.148200    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"
Jul 28 17:31:58.313983 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:58.313891    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"]
Jul 28 17:31:58.317846 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:31:58.317815    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c6f8b3_8313_4b72_a03d_0399f41c5aa8.slice/crio-1bbf4ae9b8b66da834fe963550e9ae3354d7e37092fe175203956bba49fd0e77 WatchSource:0}: Error finding container 1bbf4ae9b8b66da834fe963550e9ae3354d7e37092fe175203956bba49fd0e77: Status 404 returned error can't find the container with id 1bbf4ae9b8b66da834fe963550e9ae3354d7e37092fe175203956bba49fd0e77
Jul 28 17:31:58.402280 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:58.402219    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:58.669484 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:58.669455    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:58.909817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:31:58.909772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw" event={"ID":"60c6f8b3-8313-4b72-a03d-0399f41c5aa8","Type":"ContainerStarted","Data":"1bbf4ae9b8b66da834fe963550e9ae3354d7e37092fe175203956bba49fd0e77"}
Jul 28 17:32:00.387517 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.387478    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-r8p5c"]
Jul 28 17:32:00.397733 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.397709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c"
Jul 28 17:32:00.400001 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.399983    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nbdjt\""
Jul 28 17:32:00.403628 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.403569    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-r8p5c"]
Jul 28 17:32:00.478821 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.478788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7qr\" (UniqueName: \"kubernetes.io/projected/a34b85d0-c9c1-424c-9479-b130b8aa5d71-kube-api-access-2w7qr\") pod \"authorino-operator-56fb6f6856-r8p5c\" (UID: \"a34b85d0-c9c1-424c-9479-b130b8aa5d71\") " pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c"
Jul 28 17:32:00.579515 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.579471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7qr\" (UniqueName: \"kubernetes.io/projected/a34b85d0-c9c1-424c-9479-b130b8aa5d71-kube-api-access-2w7qr\") pod \"authorino-operator-56fb6f6856-r8p5c\" (UID: \"a34b85d0-c9c1-424c-9479-b130b8aa5d71\") " pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c"
Jul 28 17:32:00.598860 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.598827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7qr\" (UniqueName: \"kubernetes.io/projected/a34b85d0-c9c1-424c-9479-b130b8aa5d71-kube-api-access-2w7qr\") pod \"authorino-operator-56fb6f6856-r8p5c\" (UID: \"a34b85d0-c9c1-424c-9479-b130b8aa5d71\") " pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c"
Jul 28 17:32:00.711243 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.711154    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c"
Jul 28 17:32:00.871242 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.871135    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:00.931670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.931631    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-r8p5c"]
Jul 28 17:32:00.934486 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:32:00.934461    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34b85d0_c9c1_424c_9479_b130b8aa5d71.slice/crio-fa5cd9a00926e0d0326970ea17344a5865713f1da85d2dd3d0312ad99a9b6937 WatchSource:0}: Error finding container fa5cd9a00926e0d0326970ea17344a5865713f1da85d2dd3d0312ad99a9b6937: Status 404 returned error can't find the container with id fa5cd9a00926e0d0326970ea17344a5865713f1da85d2dd3d0312ad99a9b6937
Jul 28 17:32:00.966507 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:00.966376    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:01.090306 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:01.090271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:01.925817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:01.925782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c" event={"ID":"a34b85d0-c9c1-424c-9479-b130b8aa5d71","Type":"ContainerStarted","Data":"fa5cd9a00926e0d0326970ea17344a5865713f1da85d2dd3d0312ad99a9b6937"}
Jul 28 17:32:01.927688 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:01.927658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw" event={"ID":"60c6f8b3-8313-4b72-a03d-0399f41c5aa8","Type":"ContainerStarted","Data":"7326752548ec821d05b8e1eb964de658f185f3859b7734a6aa4546d03e9d76b3"}
Jul 28 17:32:01.927817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:01.927759    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"
Jul 28 17:32:01.951141 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:01.951094    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw" podStartSLOduration=2.421256368 podStartE2EDuration="4.951074461s" totalImagesPullingTime="2.529818093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:01.949364666 +0000 UTC m=+562.603859642" watchObservedRunningTime="2026-07-28 17:32:01.951074461 +0000 UTC m=+562.605569436"
Jul 28 17:32:02.341732 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:02.341700    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:02.897213 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:02.897183    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:02.933666 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:02.933631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c" event={"ID":"a34b85d0-c9c1-424c-9479-b130b8aa5d71","Type":"ContainerStarted","Data":"05e412d243b4a133baee5e0630738913458e7977869ae840df135aeae9d4f5a2"}
Jul 28 17:32:02.934177 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:02.933921    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c"
Jul 28 17:32:02.955113 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:02.955064    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c" podStartSLOduration=1.7100996739999998 podStartE2EDuration="2.955047499s" totalImagesPullingTime="1.244947825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:02.953050383 +0000 UTC m=+563.607545358" watchObservedRunningTime="2026-07-28 17:32:02.955047499 +0000 UTC m=+563.609542473"
Jul 28 17:32:03.291619 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:03.291529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:06.409776 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:06.409718    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:06.509735 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:06.509696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:06.800981 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:06.800897    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:06.901264 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:06.901228    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:07.206374 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:07.206332    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:07.274537 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:07.274507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:12.936315 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:12.936282    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-dzzbw"
Jul 28 17:32:13.459222 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:13.459182    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:13.552248 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:13.552209    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:13.940262 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:13.940230    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-r8p5c"
Jul 28 17:32:18.234858 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:18.234816    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:18.336784 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:18.336751    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:20.681766 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:20.681727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:20.780745 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:20.780705    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:27.881333 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:27.881304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:27.969333 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:27.969295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:28.087257 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:28.087218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:28.190484 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:28.190399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:28.294246 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:28.294207    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:28.365253 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:28.365218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:29.428470 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:29.428437    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:29.502890 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:29.502815    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:35.245924 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:35.245863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:35.344075 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:35.344034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:40.046772 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:40.046729    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:40.143233 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:40.143201    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:42.528867 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:42.528830    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:42.620508 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:42.620478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:46.437502 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.437466    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:46.444030 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.444003    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"]
Jul 28 17:32:46.448065 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.448046    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:32:46.450366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.450348    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 17:32:46.450495 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.450478    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-74x2r\""
Jul 28 17:32:46.459258 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.459234    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"]
Jul 28 17:32:46.518080 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.518046    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/452be049-ff5b-44f5-a0c5-0baf8f710942-config-file\") pod \"limitador-limitador-859c4d96d6-l8fmc\" (UID: \"452be049-ff5b-44f5-a0c5-0baf8f710942\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:32:46.518317 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.518294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngzm\" (UniqueName: \"kubernetes.io/projected/452be049-ff5b-44f5-a0c5-0baf8f710942-kube-api-access-zngzm\") pod \"limitador-limitador-859c4d96d6-l8fmc\" (UID: \"452be049-ff5b-44f5-a0c5-0baf8f710942\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:32:46.539417 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.539389    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"]
Jul 28 17:32:46.548507 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.548477    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:46.618777 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.618753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/452be049-ff5b-44f5-a0c5-0baf8f710942-config-file\") pod \"limitador-limitador-859c4d96d6-l8fmc\" (UID: \"452be049-ff5b-44f5-a0c5-0baf8f710942\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:32:46.618777 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.618784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngzm\" (UniqueName: \"kubernetes.io/projected/452be049-ff5b-44f5-a0c5-0baf8f710942-kube-api-access-zngzm\") pod \"limitador-limitador-859c4d96d6-l8fmc\" (UID: \"452be049-ff5b-44f5-a0c5-0baf8f710942\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:32:46.619277 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.619252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:46.619571 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.619551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/452be049-ff5b-44f5-a0c5-0baf8f710942-config-file\") pod \"limitador-limitador-859c4d96d6-l8fmc\" (UID: \"452be049-ff5b-44f5-a0c5-0baf8f710942\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:32:46.627247 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.627220    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngzm\" (UniqueName: \"kubernetes.io/projected/452be049-ff5b-44f5-a0c5-0baf8f710942-kube-api-access-zngzm\") pod \"limitador-limitador-859c4d96d6-l8fmc\" (UID: \"452be049-ff5b-44f5-a0c5-0baf8f710942\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:32:46.757933 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.757849    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:32:46.890389 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:46.890366    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"]
Jul 28 17:32:46.893337 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:32:46.893310    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452be049_ff5b_44f5_a0c5_0baf8f710942.slice/crio-86ae441b7b23802cbd31247cb0cf1fa624a817c76dbbe5b05e2a261f9f496e61 WatchSource:0}: Error finding container 86ae441b7b23802cbd31247cb0cf1fa624a817c76dbbe5b05e2a261f9f496e61: Status 404 returned error can't find the container with id 86ae441b7b23802cbd31247cb0cf1fa624a817c76dbbe5b05e2a261f9f496e61
Jul 28 17:32:47.109535 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:47.109455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc" event={"ID":"452be049-ff5b-44f5-a0c5-0baf8f710942","Type":"ContainerStarted","Data":"86ae441b7b23802cbd31247cb0cf1fa624a817c76dbbe5b05e2a261f9f496e61"}
Jul 28 17:32:47.320981 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:47.320915    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:47.421398 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:47.421360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:49.119318 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:49.119277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc" event={"ID":"452be049-ff5b-44f5-a0c5-0baf8f710942","Type":"ContainerStarted","Data":"aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd"}
Jul 28 17:32:49.119762 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:49.119402    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:32:49.135238 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:32:49.135191    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc" podStartSLOduration=1.046600907 podStartE2EDuration="3.135176815s" totalImagesPullingTime="2.088575908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:49.133794274 +0000 UTC m=+609.788289247" watchObservedRunningTime="2026-07-28 17:32:49.135176815 +0000 UTC m=+609.789671789"
Jul 28 17:33:00.124904 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.124849    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:33:00.213336 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.213298    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"]
Jul 28 17:33:00.213511 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.213479    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc" podUID="452be049-ff5b-44f5-a0c5-0baf8f710942" containerName="limitador" containerID="cri-o://aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd" gracePeriod=30
Jul 28 17:33:00.752228 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.752205    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:33:00.829173 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.829073    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zngzm\" (UniqueName: \"kubernetes.io/projected/452be049-ff5b-44f5-a0c5-0baf8f710942-kube-api-access-zngzm\") pod \"452be049-ff5b-44f5-a0c5-0baf8f710942\" (UID: \"452be049-ff5b-44f5-a0c5-0baf8f710942\") "
Jul 28 17:33:00.829173 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.829129    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/452be049-ff5b-44f5-a0c5-0baf8f710942-config-file\") pod \"452be049-ff5b-44f5-a0c5-0baf8f710942\" (UID: \"452be049-ff5b-44f5-a0c5-0baf8f710942\") "
Jul 28 17:33:00.829534 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.829507    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452be049-ff5b-44f5-a0c5-0baf8f710942-config-file" (OuterVolumeSpecName: "config-file") pod "452be049-ff5b-44f5-a0c5-0baf8f710942" (UID: "452be049-ff5b-44f5-a0c5-0baf8f710942"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:00.831400 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.831378    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452be049-ff5b-44f5-a0c5-0baf8f710942-kube-api-access-zngzm" (OuterVolumeSpecName: "kube-api-access-zngzm") pod "452be049-ff5b-44f5-a0c5-0baf8f710942" (UID: "452be049-ff5b-44f5-a0c5-0baf8f710942"). InnerVolumeSpecName "kube-api-access-zngzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:00.930621 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.930589    2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/452be049-ff5b-44f5-a0c5-0baf8f710942-config-file\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:33:00.930621 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:00.930615    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngzm\" (UniqueName: \"kubernetes.io/projected/452be049-ff5b-44f5-a0c5-0baf8f710942-kube-api-access-zngzm\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:33:01.169427 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.169391    2559 generic.go:356] "Generic (PLEG): container finished" podID="452be049-ff5b-44f5-a0c5-0baf8f710942" containerID="aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd" exitCode=0
Jul 28 17:33:01.169813 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.169453    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"
Jul 28 17:33:01.169813 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.169481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc" event={"ID":"452be049-ff5b-44f5-a0c5-0baf8f710942","Type":"ContainerDied","Data":"aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd"}
Jul 28 17:33:01.169813 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.169537    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-l8fmc" event={"ID":"452be049-ff5b-44f5-a0c5-0baf8f710942","Type":"ContainerDied","Data":"86ae441b7b23802cbd31247cb0cf1fa624a817c76dbbe5b05e2a261f9f496e61"}
Jul 28 17:33:01.169813 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.169558    2559 scope.go:117] "RemoveContainer" containerID="aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd"
Jul 28 17:33:01.182625 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.182602    2559 scope.go:117] "RemoveContainer" containerID="aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd"
Jul 28 17:33:01.183066 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:33:01.183032    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd\": container with ID starting with aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd not found: ID does not exist" containerID="aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd"
Jul 28 17:33:01.183172 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.183077    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd"} err="failed to get container status \"aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd\": rpc error: code = NotFound desc = could not find container \"aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd\": container with ID starting with aaa13e2a87e7e308ed93d73c5ae6549ccd8e6f7ea5f6bd851dcf364fd26060fd not found: ID does not exist"
Jul 28 17:33:01.192824 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.192787    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"]
Jul 28 17:33:01.195906 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.195869    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-l8fmc"]
Jul 28 17:33:01.208917 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:33:01.208887    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452be049_ff5b_44f5_a0c5_0baf8f710942.slice\": RecentStats: unable to find data in memory cache]"
Jul 28 17:33:01.832826 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:01.832790    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="452be049-ff5b-44f5-a0c5-0baf8f710942" path="/var/lib/kubelet/pods/452be049-ff5b-44f5-a0c5-0baf8f710942/volumes"
Jul 28 17:33:06.335866 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:06.335837    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:06.527722 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:06.527683    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:17.612289 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:17.612251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:17.717719 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:17.717682    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:34.476042 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:34.476006    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:55.894473 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.894441    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:55.938589 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.938558    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-hwv4w"]
Jul 28 17:33:55.938847 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.938836    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452be049-ff5b-44f5-a0c5-0baf8f710942" containerName="limitador"
Jul 28 17:33:55.938914 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.938849    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="452be049-ff5b-44f5-a0c5-0baf8f710942" containerName="limitador"
Jul 28 17:33:55.938914 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.938912    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="452be049-ff5b-44f5-a0c5-0baf8f710942" containerName="limitador"
Jul 28 17:33:55.942936 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.942914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:33:55.945429 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.945404    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-jlnd8\""
Jul 28 17:33:55.945566 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.945547    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:33:55.953121 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.953092    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-hwv4w"]
Jul 28 17:33:55.956357 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.956339    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-t2bz8"]
Jul 28 17:33:55.961716 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.961692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:33:55.964732 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.964711    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 17:33:55.964926 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.964910    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-sqjd8\""
Jul 28 17:33:55.970284 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.970259    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-t2bz8"]
Jul 28 17:33:55.972897 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.972851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc8nz\" (UniqueName: \"kubernetes.io/projected/6fe48c5b-b209-4020-bce4-dcd8406038a9-kube-api-access-rc8nz\") pod \"model-serving-api-7b954ffc95-hwv4w\" (UID: \"6fe48c5b-b209-4020-bce4-dcd8406038a9\") " pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:33:55.972999 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.972914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fe48c5b-b209-4020-bce4-dcd8406038a9-tls-certs\") pod \"model-serving-api-7b954ffc95-hwv4w\" (UID: \"6fe48c5b-b209-4020-bce4-dcd8406038a9\") " pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:33:55.972999 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.972946    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5tv\" (UniqueName: \"kubernetes.io/projected/6a305e10-351c-48eb-95af-1dd929296f5a-kube-api-access-sq5tv\") pod \"odh-model-controller-75d9cdbddf-t2bz8\" (UID: \"6a305e10-351c-48eb-95af-1dd929296f5a\") " pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:33:55.973092 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.973003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a305e10-351c-48eb-95af-1dd929296f5a-cert\") pod \"odh-model-controller-75d9cdbddf-t2bz8\" (UID: \"6a305e10-351c-48eb-95af-1dd929296f5a\") " pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:33:55.995072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:55.995041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:56.073975 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.073947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc8nz\" (UniqueName: \"kubernetes.io/projected/6fe48c5b-b209-4020-bce4-dcd8406038a9-kube-api-access-rc8nz\") pod \"model-serving-api-7b954ffc95-hwv4w\" (UID: \"6fe48c5b-b209-4020-bce4-dcd8406038a9\") " pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:33:56.074153 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.073992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fe48c5b-b209-4020-bce4-dcd8406038a9-tls-certs\") pod \"model-serving-api-7b954ffc95-hwv4w\" (UID: \"6fe48c5b-b209-4020-bce4-dcd8406038a9\") " pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:33:56.074153 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.074021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5tv\" (UniqueName: \"kubernetes.io/projected/6a305e10-351c-48eb-95af-1dd929296f5a-kube-api-access-sq5tv\") pod \"odh-model-controller-75d9cdbddf-t2bz8\" (UID: \"6a305e10-351c-48eb-95af-1dd929296f5a\") " pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:33:56.074153 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:33:56.074106    2559 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 17:33:56.074153 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.074077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a305e10-351c-48eb-95af-1dd929296f5a-cert\") pod \"odh-model-controller-75d9cdbddf-t2bz8\" (UID: \"6a305e10-351c-48eb-95af-1dd929296f5a\") " pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:33:56.074153 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:33:56.074140    2559 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 17:33:56.074419 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:33:56.074178    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe48c5b-b209-4020-bce4-dcd8406038a9-tls-certs podName:6fe48c5b-b209-4020-bce4-dcd8406038a9 nodeName:}" failed. No retries permitted until 2026-07-28 17:33:56.574156593 +0000 UTC m=+677.228651549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6fe48c5b-b209-4020-bce4-dcd8406038a9-tls-certs") pod "model-serving-api-7b954ffc95-hwv4w" (UID: "6fe48c5b-b209-4020-bce4-dcd8406038a9") : secret "model-serving-api-tls" not found
Jul 28 17:33:56.074419 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:33:56.074222    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a305e10-351c-48eb-95af-1dd929296f5a-cert podName:6a305e10-351c-48eb-95af-1dd929296f5a nodeName:}" failed. No retries permitted until 2026-07-28 17:33:56.574202181 +0000 UTC m=+677.228697136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6a305e10-351c-48eb-95af-1dd929296f5a-cert") pod "odh-model-controller-75d9cdbddf-t2bz8" (UID: "6a305e10-351c-48eb-95af-1dd929296f5a") : secret "odh-model-controller-webhook-cert" not found
Jul 28 17:33:56.082696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.082673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc8nz\" (UniqueName: \"kubernetes.io/projected/6fe48c5b-b209-4020-bce4-dcd8406038a9-kube-api-access-rc8nz\") pod \"model-serving-api-7b954ffc95-hwv4w\" (UID: \"6fe48c5b-b209-4020-bce4-dcd8406038a9\") " pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:33:56.083264 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.083247    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5tv\" (UniqueName: \"kubernetes.io/projected/6a305e10-351c-48eb-95af-1dd929296f5a-kube-api-access-sq5tv\") pod \"odh-model-controller-75d9cdbddf-t2bz8\" (UID: \"6a305e10-351c-48eb-95af-1dd929296f5a\") " pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:33:56.193948 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.193818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:56.334311 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.334258    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:56.578673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.578584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fe48c5b-b209-4020-bce4-dcd8406038a9-tls-certs\") pod \"model-serving-api-7b954ffc95-hwv4w\" (UID: \"6fe48c5b-b209-4020-bce4-dcd8406038a9\") " pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:33:56.578673 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.578647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a305e10-351c-48eb-95af-1dd929296f5a-cert\") pod \"odh-model-controller-75d9cdbddf-t2bz8\" (UID: \"6a305e10-351c-48eb-95af-1dd929296f5a\") " pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:33:56.581198 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.581167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a305e10-351c-48eb-95af-1dd929296f5a-cert\") pod \"odh-model-controller-75d9cdbddf-t2bz8\" (UID: \"6a305e10-351c-48eb-95af-1dd929296f5a\") " pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:33:56.581315 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.581294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fe48c5b-b209-4020-bce4-dcd8406038a9-tls-certs\") pod \"model-serving-api-7b954ffc95-hwv4w\" (UID: \"6fe48c5b-b209-4020-bce4-dcd8406038a9\") " pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:33:56.701266 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.701226    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:56.763121 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.763091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:56.854133 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.854057    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:33:56.873377 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.873353    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:33:56.991804 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.991775    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-hwv4w"]
Jul 28 17:33:56.995355 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:33:56.995329    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe48c5b_b209_4020_bce4_dcd8406038a9.slice/crio-fafa4071c55eb56a8624343aacf04d19f5f0575946e616e49eee5bedf01cceae WatchSource:0}: Error finding container fafa4071c55eb56a8624343aacf04d19f5f0575946e616e49eee5bedf01cceae: Status 404 returned error can't find the container with id fafa4071c55eb56a8624343aacf04d19f5f0575946e616e49eee5bedf01cceae
Jul 28 17:33:56.997703 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:56.997685    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:33:57.018138 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:57.018111    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-t2bz8"]
Jul 28 17:33:57.020673 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:33:57.020650    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a305e10_351c_48eb_95af_1dd929296f5a.slice/crio-bbec959fdf21ac84bc11a82f5b91db32663be7991993f7bf1862d312e7122b7f WatchSource:0}: Error finding container bbec959fdf21ac84bc11a82f5b91db32663be7991993f7bf1862d312e7122b7f: Status 404 returned error can't find the container with id bbec959fdf21ac84bc11a82f5b91db32663be7991993f7bf1862d312e7122b7f
Jul 28 17:33:57.394137 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:57.394102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-t2bz8" event={"ID":"6a305e10-351c-48eb-95af-1dd929296f5a","Type":"ContainerStarted","Data":"bbec959fdf21ac84bc11a82f5b91db32663be7991993f7bf1862d312e7122b7f"}
Jul 28 17:33:57.395202 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:33:57.395177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-hwv4w" event={"ID":"6fe48c5b-b209-4020-bce4-dcd8406038a9","Type":"ContainerStarted","Data":"fafa4071c55eb56a8624343aacf04d19f5f0575946e616e49eee5bedf01cceae"}
Jul 28 17:34:01.419718 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:01.419679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-t2bz8" event={"ID":"6a305e10-351c-48eb-95af-1dd929296f5a","Type":"ContainerStarted","Data":"b21f2395333d19e053251463876399a19a5fc5478e57a5f1e135464bffeba3bf"}
Jul 28 17:34:01.420217 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:01.419798    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:34:01.421716 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:01.421691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-hwv4w" event={"ID":"6fe48c5b-b209-4020-bce4-dcd8406038a9","Type":"ContainerStarted","Data":"9ebca912fb74bd6f828e4699c802fe123dabd4045c66192120fa20eff8f0f7b4"}
Jul 28 17:34:01.421828 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:01.421757    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:34:01.436209 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:01.436154    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-t2bz8" podStartSLOduration=2.755014081 podStartE2EDuration="6.436134733s" totalImagesPullingTime="3.681120652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:01.435153051 +0000 UTC m=+682.089648027" watchObservedRunningTime="2026-07-28 17:34:01.436134733 +0000 UTC m=+682.090629689"
Jul 28 17:34:01.458964 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:01.458908    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-hwv4w" podStartSLOduration=2.715156356 podStartE2EDuration="6.458866681s" totalImagesPullingTime="3.743710325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:01.457415244 +0000 UTC m=+682.111910219" watchObservedRunningTime="2026-07-28 17:34:01.458866681 +0000 UTC m=+682.113361655"
Jul 28 17:34:12.428857 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:12.428825    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-t2bz8"
Jul 28 17:34:12.430757 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:12.430735    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-hwv4w"
Jul 28 17:34:12.694001 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:12.693919    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:22.412255 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:22.412221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:22.467089 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:22.467051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:35.895931 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:35.895893    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:35.954241 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:34:35.954212    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:30.390057 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:30.390025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:30.462431 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:30.462400    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:30.527113 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:30.527074    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:31.111124 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:31.111087    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:31.213203 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:31.213167    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:31.990615 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:31.990568    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:32.089828 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:32.089792    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:41.181870 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:41.181836    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:41.284069 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:41.284035    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:41.345572 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:41.345530    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:44.918539 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:44.918503    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:45.045303 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:35:45.045267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:07.943051 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:07.942942    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:13.135943 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.135908    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:13.538226 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.538146    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"]
Jul 28 17:36:13.541327 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.541303    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:13.545811 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.545784    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 28 17:36:13.545959 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.545805    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 28 17:36:13.545959 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.545792    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-6tjbp\""
Jul 28 17:36:13.545959 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.545885    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-dcvlk\""
Jul 28 17:36:13.546132 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.545987    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 28 17:36:13.553993 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.553971    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"]
Jul 28 17:36:13.580721 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.580671    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:13.618902 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.618274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gz5\" (UniqueName: \"kubernetes.io/projected/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-kube-api-access-z4gz5\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds\" (UID: \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:13.618902 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.618392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds\" (UID: \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:13.720223 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.719668    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds\" (UID: \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:13.720419 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.720297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gz5\" (UniqueName: \"kubernetes.io/projected/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-kube-api-access-z4gz5\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds\" (UID: \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:13.723355 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.723322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds\" (UID: \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:13.730048 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.730027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gz5\" (UniqueName: \"kubernetes.io/projected/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-kube-api-access-z4gz5\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds\" (UID: \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:13.851972 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:13.851864    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:14.018219 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:14.018121    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"]
Jul 28 17:36:14.026225 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:36:14.026183    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecee318_2f96_4fba_8e55_66f81b4c1cd6.slice/crio-e03bde314bb76e33bc9f0ed29bf9e62edf19260f01479a705067c6accfa81ddd WatchSource:0}: Error finding container e03bde314bb76e33bc9f0ed29bf9e62edf19260f01479a705067c6accfa81ddd: Status 404 returned error can't find the container with id e03bde314bb76e33bc9f0ed29bf9e62edf19260f01479a705067c6accfa81ddd
Jul 28 17:36:14.986200 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:14.986115    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds" event={"ID":"5ecee318-2f96-4fba-8e55-66f81b4c1cd6","Type":"ContainerStarted","Data":"e03bde314bb76e33bc9f0ed29bf9e62edf19260f01479a705067c6accfa81ddd"}
Jul 28 17:36:16.997354 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:16.997315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds" event={"ID":"5ecee318-2f96-4fba-8e55-66f81b4c1cd6","Type":"ContainerStarted","Data":"37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00"}
Jul 28 17:36:16.997778 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:16.997399    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:17.018475 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:17.018423    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds" podStartSLOduration=1.642733622 podStartE2EDuration="4.01840839s" totalImagesPullingTime="2.375674768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:17.016080834 +0000 UTC m=+817.670575808" watchObservedRunningTime="2026-07-28 17:36:17.01840839 +0000 UTC m=+817.672903365"
Jul 28 17:36:18.003240 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:18.003207    2559 generic.go:356] "Generic (PLEG): container finished" podID="67fbff38-b30c-4641-95cc-f266c7219170" containerID="a1498e0fc22636abe80ef24ac58ebeb1137063bb6532069f34dbc84e07da5ae5" exitCode=0
Jul 28 17:36:18.003636 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:18.003282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" event={"ID":"67fbff38-b30c-4641-95cc-f266c7219170","Type":"ContainerDied","Data":"a1498e0fc22636abe80ef24ac58ebeb1137063bb6532069f34dbc84e07da5ae5"}
Jul 28 17:36:18.003794 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:18.003772    2559 scope.go:117] "RemoveContainer" containerID="a1498e0fc22636abe80ef24ac58ebeb1137063bb6532069f34dbc84e07da5ae5"
Jul 28 17:36:18.004821 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:18.004794    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:36:18.004910 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:18.004858    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:36:18.005110 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:18.004914    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:36:19.013596 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:19.013531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" event={"ID":"67fbff38-b30c-4641-95cc-f266c7219170","Type":"ContainerStarted","Data":"aede967a08b7660bafe90cc0db6eb50783ab8e4bd12d1e3cf044abdc79367a04"}
Jul 28 17:36:19.615944 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:19.615904    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:36:19.616892 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:19.616841    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused" start-of-body=
Jul 28 17:36:19.617004 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:19.616922    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" podUID="67fbff38-b30c-4641-95cc-f266c7219170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused"
Jul 28 17:36:20.616001 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:20.615961    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused" start-of-body=
Jul 28 17:36:20.616488 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:20.616034    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" podUID="67fbff38-b30c-4641-95cc-f266c7219170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused"
Jul 28 17:36:21.615425 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:21.615372    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused" start-of-body=
Jul 28 17:36:21.615628 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:21.615446    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" podUID="67fbff38-b30c-4641-95cc-f266c7219170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused"
Jul 28 17:36:22.615372 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:22.615334    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused" start-of-body=
Jul 28 17:36:22.615958 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:22.615925    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" podUID="67fbff38-b30c-4641-95cc-f266c7219170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused"
Jul 28 17:36:22.830962 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:22.830927    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:36:23.615453 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:23.615414    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused" start-of-body=
Jul 28 17:36:23.616067 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:23.616033    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg" podUID="67fbff38-b30c-4641-95cc-f266c7219170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.19:15021/healthz/ready\": dial tcp 10.132.0.19:15021: connect: connection refused"
Jul 28 17:36:24.800962 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:24.800916    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:36:24.803459 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:24.803427    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg"
Jul 28 17:36:39.356213 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:39.356172    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:48.005530 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:48.005499    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:36:54.430833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:54.430806    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"]
Jul 28 17:36:54.432657 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:54.432610    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:54.528180 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:54.528139    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:58.005275 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:36:58.005241    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds" podUID="5ecee318-2f96-4fba-8e55-66f81b4c1cd6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:37:08.004921 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:08.004858    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds" podUID="5ecee318-2f96-4fba-8e55-66f81b4c1cd6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:37:09.451105 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:09.451043    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds" podUID="5ecee318-2f96-4fba-8e55-66f81b4c1cd6" containerName="main" containerID="cri-o://37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00" gracePeriod=45
Jul 28 17:37:09.704336 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:09.704278    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:37:09.781264 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:09.781222    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4gz5\" (UniqueName: \"kubernetes.io/projected/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-kube-api-access-z4gz5\") pod \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\" (UID: \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\") "
Jul 28 17:37:09.781454 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:09.781361    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-tls-certs\") pod \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\" (UID: \"5ecee318-2f96-4fba-8e55-66f81b4c1cd6\") "
Jul 28 17:37:09.783600 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:09.783574    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-kube-api-access-z4gz5" (OuterVolumeSpecName: "kube-api-access-z4gz5") pod "5ecee318-2f96-4fba-8e55-66f81b4c1cd6" (UID: "5ecee318-2f96-4fba-8e55-66f81b4c1cd6"). InnerVolumeSpecName "kube-api-access-z4gz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:09.783718 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:09.783615    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ecee318-2f96-4fba-8e55-66f81b4c1cd6" (UID: "5ecee318-2f96-4fba-8e55-66f81b4c1cd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:37:09.882452 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:09.882418    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:37:09.882452 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:09.882450    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4gz5\" (UniqueName: \"kubernetes.io/projected/5ecee318-2f96-4fba-8e55-66f81b4c1cd6-kube-api-access-z4gz5\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:37:10.244991 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:10.244955    2559 generic.go:356] "Generic (PLEG): container finished" podID="5ecee318-2f96-4fba-8e55-66f81b4c1cd6" containerID="37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00" exitCode=0
Jul 28 17:37:10.245194 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:10.245020    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"
Jul 28 17:37:10.245194 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:10.245044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds" event={"ID":"5ecee318-2f96-4fba-8e55-66f81b4c1cd6","Type":"ContainerDied","Data":"37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00"}
Jul 28 17:37:10.245194 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:10.245129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds" event={"ID":"5ecee318-2f96-4fba-8e55-66f81b4c1cd6","Type":"ContainerDied","Data":"e03bde314bb76e33bc9f0ed29bf9e62edf19260f01479a705067c6accfa81ddd"}
Jul 28 17:37:10.245194 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:10.245152    2559 scope.go:117] "RemoveContainer" containerID="37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00"
Jul 28 17:37:10.253219 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:10.253201    2559 scope.go:117] "RemoveContainer" containerID="37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00"
Jul 28 17:37:10.253505 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:37:10.253486    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00\": container with ID starting with 37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00 not found: ID does not exist" containerID="37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00"
Jul 28 17:37:10.253548 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:10.253515    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00"} err="failed to get container status \"37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00\": rpc error: code = NotFound desc = could not find container \"37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00\": container with ID starting with 37bd7634b836bda0755ad4cf20cea8e909f422475264294cb5eadc5749d79c00 not found: ID does not exist"
Jul 28 17:37:10.263269 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:10.263240    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"]
Jul 28 17:37:10.267440 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:10.267418    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-st4ds"]
Jul 28 17:37:11.832965 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:11.832935    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ecee318-2f96-4fba-8e55-66f81b4c1cd6" path="/var/lib/kubelet/pods/5ecee318-2f96-4fba-8e55-66f81b4c1cd6/volumes"
Jul 28 17:37:14.465389 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:14.465347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:14.572323 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:14.572262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:21.289807 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:21.289773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:22.298342 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.298309    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:22.539151 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.539115    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"]
Jul 28 17:37:22.539512 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.539490    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecee318-2f96-4fba-8e55-66f81b4c1cd6" containerName="main"
Jul 28 17:37:22.539512 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.539513    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecee318-2f96-4fba-8e55-66f81b4c1cd6" containerName="main"
Jul 28 17:37:22.539674 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.539569    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecee318-2f96-4fba-8e55-66f81b4c1cd6" containerName="main"
Jul 28 17:37:22.543978 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.543954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:37:22.546576 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.546556    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 28 17:37:22.547392 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.547361    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-mt4zr\""
Jul 28 17:37:22.547497 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.547364    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-j6vh6\""
Jul 28 17:37:22.547497 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.547365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 28 17:37:22.547591 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.547373    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 28 17:37:22.551702 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.551645    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"]
Jul 28 17:37:22.572255 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.572221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:22.687044 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.687008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzmhx\" (UniqueName: \"kubernetes.io/projected/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-kube-api-access-vzmhx\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68\" (UID: \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:37:22.687210 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.687069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68\" (UID: \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:37:22.788378 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.788340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzmhx\" (UniqueName: \"kubernetes.io/projected/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-kube-api-access-vzmhx\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68\" (UID: \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:37:22.788557 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.788394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68\" (UID: \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:37:22.790954 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.790929    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68\" (UID: \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:37:22.796862 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.796839    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzmhx\" (UniqueName: \"kubernetes.io/projected/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-kube-api-access-vzmhx\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68\" (UID: \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:37:22.858095 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.858005    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:37:22.990818 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:22.990790    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"]
Jul 28 17:37:22.993663 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:37:22.993637    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c10dc1_b3e0_422a_b657_6e0cb4d7f443.slice/crio-fe58ebe0ec4122d9fd9cf24a055835508d331822ef99f54fabe6a3a4757cfe11 WatchSource:0}: Error finding container fe58ebe0ec4122d9fd9cf24a055835508d331822ef99f54fabe6a3a4757cfe11: Status 404 returned error can't find the container with id fe58ebe0ec4122d9fd9cf24a055835508d331822ef99f54fabe6a3a4757cfe11
Jul 28 17:37:23.301075 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:23.301043    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68" event={"ID":"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443","Type":"ContainerStarted","Data":"919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f"}
Jul 28 17:37:23.301476 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:23.301092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68" event={"ID":"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443","Type":"ContainerStarted","Data":"fe58ebe0ec4122d9fd9cf24a055835508d331822ef99f54fabe6a3a4757cfe11"}
Jul 28 17:37:23.301476 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:23.301129    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:37:23.327423 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:23.327356    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68" podStartSLOduration=1.327333793 podStartE2EDuration="1.327333793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:23.325028135 +0000 UTC m=+883.979523111" watchObservedRunningTime="2026-07-28 17:37:23.327333793 +0000 UTC m=+883.981828767"
Jul 28 17:37:26.408458 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:26.408415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:26.468626 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:26.468596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:27.334496 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:27.334457    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:27.434537 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:27.434504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:32.005514 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:32.005478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:54.307482 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:37:54.307453    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:38:01.166228 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:01.166179    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:01.174437 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:01.174409    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"]
Jul 28 17:38:01.241851 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:01.241813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:04.306809 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:04.306777    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68" podUID="b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:38:14.307724 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:14.307686    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68" podUID="b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:38:15.629318 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:15.629280    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:16.194061 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.194020    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68" podUID="b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" containerName="main" containerID="cri-o://919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f" gracePeriod=45
Jul 28 17:38:16.436182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.436159    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:38:16.532401 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.532314    2559 generic.go:356] "Generic (PLEG): container finished" podID="b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" containerID="919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f" exitCode=0
Jul 28 17:38:16.532401 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.532360    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"
Jul 28 17:38:16.532401 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.532367    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68" event={"ID":"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443","Type":"ContainerDied","Data":"919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f"}
Jul 28 17:38:16.532646 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.532408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68" event={"ID":"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443","Type":"ContainerDied","Data":"fe58ebe0ec4122d9fd9cf24a055835508d331822ef99f54fabe6a3a4757cfe11"}
Jul 28 17:38:16.532646 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.532431    2559 scope.go:117] "RemoveContainer" containerID="919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f"
Jul 28 17:38:16.539915 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.539862    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-tls-certs\") pod \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\" (UID: \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\") "
Jul 28 17:38:16.540014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.539953    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzmhx\" (UniqueName: \"kubernetes.io/projected/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-kube-api-access-vzmhx\") pod \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\" (UID: \"b4c10dc1-b3e0-422a-b657-6e0cb4d7f443\") "
Jul 28 17:38:16.540466 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.540443    2559 scope.go:117] "RemoveContainer" containerID="919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f"
Jul 28 17:38:16.540934 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:38:16.540913    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f\": container with ID starting with 919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f not found: ID does not exist" containerID="919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f"
Jul 28 17:38:16.541029 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.540940    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f"} err="failed to get container status \"919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f\": rpc error: code = NotFound desc = could not find container \"919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f\": container with ID starting with 919c8168a653e9dc781f94b6483bc508331ff6b5aa0848479df65ef1b304ee2f not found: ID does not exist"
Jul 28 17:38:16.542175 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.542155    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" (UID: "b4c10dc1-b3e0-422a-b657-6e0cb4d7f443"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:16.542258 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.542175    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-kube-api-access-vzmhx" (OuterVolumeSpecName: "kube-api-access-vzmhx") pod "b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" (UID: "b4c10dc1-b3e0-422a-b657-6e0cb4d7f443"). InnerVolumeSpecName "kube-api-access-vzmhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:16.640830 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.640791    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzmhx\" (UniqueName: \"kubernetes.io/projected/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-kube-api-access-vzmhx\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:16.640830 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.640829    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:16.853222 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.853192    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"]
Jul 28 17:38:16.857997 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:16.857965    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-hgq68"]
Jul 28 17:38:17.833011 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:17.832978    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" path="/var/lib/kubelet/pods/b4c10dc1-b3e0-422a-b657-6e0cb4d7f443/volumes"
Jul 28 17:38:21.181516 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.181474    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:21.283519 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.283468    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:21.387414 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.387379    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:21.449660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.449589    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"]
Jul 28 17:38:21.449987 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.449974    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" containerName="main"
Jul 28 17:38:21.450036 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.449989    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" containerName="main"
Jul 28 17:38:21.450070 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.450051    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c10dc1-b3e0-422a-b657-6e0cb4d7f443" containerName="main"
Jul 28 17:38:21.454334 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.454315    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.456775 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.456749    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 28 17:38:21.456928 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.456812    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 28 17:38:21.457107 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.457091    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 28 17:38:21.457178 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.457161    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-qn8dp\""
Jul 28 17:38:21.469705 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.469675    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"]
Jul 28 17:38:21.486670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.486639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:21.580106 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.580075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.580326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.580112    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b753-468e-4702-8737-4e88df43eec2-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.580326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.580139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.580326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.580168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b753-468e-4702-8737-4e88df43eec2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.580326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.580207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.580326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.580231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.580326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.580247    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lzhr\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-kube-api-access-5lzhr\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.580326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.580274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.580326 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.580299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.588761 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.588732    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:21.681575 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.681532    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.681575 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.681585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.681836 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.681604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b753-468e-4702-8737-4e88df43eec2-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.681836 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.681626    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.681836 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.681641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b753-468e-4702-8737-4e88df43eec2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.681836 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.681666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.681836 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.681698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.681836 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.681717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lzhr\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-kube-api-access-5lzhr\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.681836 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.681734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.682200 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.682096    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.682200 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.682175    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.682345 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.682321    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.682345 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.682336    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.682675 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.682431    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b753-468e-4702-8737-4e88df43eec2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.684182 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.684156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.684331 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.684315    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b753-468e-4702-8737-4e88df43eec2-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.689716 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.689691    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.689820 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.689737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lzhr\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-kube-api-access-5lzhr\") pod \"router-gateway-1-openshift-default-5b4d9458bd-x8kkc\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.765074 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.764955    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:21.905910 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.905858    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"]
Jul 28 17:38:21.907845 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:38:21.907814    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f2b753_468e_4702_8737_4e88df43eec2.slice/crio-0e572c5818672a5c057cb41ae379b38944204216f2218b16352808b13e09da98 WatchSource:0}: Error finding container 0e572c5818672a5c057cb41ae379b38944204216f2218b16352808b13e09da98: Status 404 returned error can't find the container with id 0e572c5818672a5c057cb41ae379b38944204216f2218b16352808b13e09da98
Jul 28 17:38:21.909964 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.909927    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:38:21.910044 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.910011    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:38:21.910083 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:21.910051    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:38:22.561895 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:22.561843    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc" event={"ID":"d4f2b753-468e-4702-8737-4e88df43eec2","Type":"ContainerStarted","Data":"9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd"}
Jul 28 17:38:22.562276 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:22.561930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc" event={"ID":"d4f2b753-468e-4702-8737-4e88df43eec2","Type":"ContainerStarted","Data":"0e572c5818672a5c057cb41ae379b38944204216f2218b16352808b13e09da98"}
Jul 28 17:38:22.585244 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:22.585025    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc" podStartSLOduration=1.585005277 podStartE2EDuration="1.585005277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:22.582731751 +0000 UTC m=+943.237226763" watchObservedRunningTime="2026-07-28 17:38:22.585005277 +0000 UTC m=+943.239500254"
Jul 28 17:38:22.765094 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:22.765059    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:22.770056 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:22.770030    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:23.566793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:23.566764    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:23.567730 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:23.567712    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:25.327027 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:25.326995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:27.913193 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:27.913162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:29.521459 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:29.521427    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:31.728482 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:31.728449    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:31.840457 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:31.840421    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:31.938518 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:31.938478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:32.019384 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.019292    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"]
Jul 28 17:38:32.023604 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.023581    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.029184 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.029156    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 28 17:38:32.031223 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.031166    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-thzvs\""
Jul 28 17:38:32.031435 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.031414    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 28 17:38:32.031550 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.031480    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 28 17:38:32.043307 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.043253    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:32.044324 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.044305    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"]
Jul 28 17:38:32.139150 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.139111    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:32.175167 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.175129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.175427 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.175194    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.175427 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.175249    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe45dab-6862-4114-8686-eb08526f2e1d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.175427 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.175307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.175427 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.175333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.175427 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.175382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.175427 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.175408    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.175864 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.175454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpfcd\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-kube-api-access-bpfcd\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.175864 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.175488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276455 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfcd\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-kube-api-access-bpfcd\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276455 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276455 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276421    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe45dab-6862-4114-8686-eb08526f2e1d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276512    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.276907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.277206 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.276984    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.277206 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.277059    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.277206 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.277100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.277299 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.277268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe45dab-6862-4114-8686-eb08526f2e1d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.278891 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.278857    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.279120 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.279101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.284386 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.284361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.284482 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.284412    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpfcd\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-kube-api-access-bpfcd\") pod \"router-gateway-1-openshift-default-66bbccc5fb-czpch\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.340212 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.340171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:32.565670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.565588    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"]
Jul 28 17:38:32.568462 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:38:32.568430    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe45dab_6862_4114_8686_eb08526f2e1d.slice/crio-760e94800da6ea03a980ada8548cae761f1bd99c76bea19aa5a0130d5b47c171 WatchSource:0}: Error finding container 760e94800da6ea03a980ada8548cae761f1bd99c76bea19aa5a0130d5b47c171: Status 404 returned error can't find the container with id 760e94800da6ea03a980ada8548cae761f1bd99c76bea19aa5a0130d5b47c171
Jul 28 17:38:32.570417 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.570380    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:38:32.570505 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.570488    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:38:32.570548 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.570532    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 17:38:32.624429 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:32.624312    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch" event={"ID":"0fe45dab-6862-4114-8686-eb08526f2e1d","Type":"ContainerStarted","Data":"760e94800da6ea03a980ada8548cae761f1bd99c76bea19aa5a0130d5b47c171"}
Jul 28 17:38:33.630532 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:33.630492    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch" event={"ID":"0fe45dab-6862-4114-8686-eb08526f2e1d","Type":"ContainerStarted","Data":"38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2"}
Jul 28 17:38:33.650856 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:33.650802    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch" podStartSLOduration=2.65078436 podStartE2EDuration="2.65078436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:33.650082188 +0000 UTC m=+954.304577164" watchObservedRunningTime="2026-07-28 17:38:33.65078436 +0000 UTC m=+954.305279334"
Jul 28 17:38:34.340364 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:34.340325    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:34.345717 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:34.345686    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:34.634938 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:34.634910    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:34.636014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:34.635992    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:34.973746 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:34.973636    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:35.089325 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:35.089294    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:35.196669 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:35.196634    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:36.714236 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:36.714205    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"]
Jul 28 17:38:36.714607 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:36.714412    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc" podUID="d4f2b753-468e-4702-8737-4e88df43eec2" containerName="istio-proxy" containerID="cri-o://9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd" gracePeriod=30
Jul 28 17:38:36.807991 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:36.807940    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:39.609102 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.609053    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc" podUID="d4f2b753-468e-4702-8737-4e88df43eec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:38:39.864610 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.864590    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:39.945814 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.945785    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-istio-token\") pod \"d4f2b753-468e-4702-8737-4e88df43eec2\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") "
Jul 28 17:38:39.946014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.945822    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-data\") pod \"d4f2b753-468e-4702-8737-4e88df43eec2\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") "
Jul 28 17:38:39.946014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.945855    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-envoy\") pod \"d4f2b753-468e-4702-8737-4e88df43eec2\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") "
Jul 28 17:38:39.946014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.945887    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lzhr\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-kube-api-access-5lzhr\") pod \"d4f2b753-468e-4702-8737-4e88df43eec2\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") "
Jul 28 17:38:39.946014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.945905    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-certs\") pod \"d4f2b753-468e-4702-8737-4e88df43eec2\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") "
Jul 28 17:38:39.946014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.945954    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b753-468e-4702-8737-4e88df43eec2-istiod-ca-cert\") pod \"d4f2b753-468e-4702-8737-4e88df43eec2\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") "
Jul 28 17:38:39.946014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.945988    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-socket\") pod \"d4f2b753-468e-4702-8737-4e88df43eec2\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") "
Jul 28 17:38:39.946333 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.946055    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b753-468e-4702-8737-4e88df43eec2-istio-podinfo\") pod \"d4f2b753-468e-4702-8737-4e88df43eec2\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") "
Jul 28 17:38:39.946333 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.946104    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-credential-socket\") pod \"d4f2b753-468e-4702-8737-4e88df43eec2\" (UID: \"d4f2b753-468e-4702-8737-4e88df43eec2\") "
Jul 28 17:38:39.946333 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.946139    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4f2b753-468e-4702-8737-4e88df43eec2" (UID: "d4f2b753-468e-4702-8737-4e88df43eec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:39.946333 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.946274    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4f2b753-468e-4702-8737-4e88df43eec2" (UID: "d4f2b753-468e-4702-8737-4e88df43eec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:39.946699 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.946349    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-data\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:39.946699 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.946367    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-socket\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:39.946699 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.946464    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4f2b753-468e-4702-8737-4e88df43eec2" (UID: "d4f2b753-468e-4702-8737-4e88df43eec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:39.946699 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.946628    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4f2b753-468e-4702-8737-4e88df43eec2" (UID: "d4f2b753-468e-4702-8737-4e88df43eec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:39.946945 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.946809    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f2b753-468e-4702-8737-4e88df43eec2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4f2b753-468e-4702-8737-4e88df43eec2" (UID: "d4f2b753-468e-4702-8737-4e88df43eec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:39.948494 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.948446    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4f2b753-468e-4702-8737-4e88df43eec2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4f2b753-468e-4702-8737-4e88df43eec2" (UID: "d4f2b753-468e-4702-8737-4e88df43eec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 17:38:39.948616 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.948579    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4f2b753-468e-4702-8737-4e88df43eec2" (UID: "d4f2b753-468e-4702-8737-4e88df43eec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:39.948726 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.948706    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4f2b753-468e-4702-8737-4e88df43eec2" (UID: "d4f2b753-468e-4702-8737-4e88df43eec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:39.948778 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:39.948704    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-kube-api-access-5lzhr" (OuterVolumeSpecName: "kube-api-access-5lzhr") pod "d4f2b753-468e-4702-8737-4e88df43eec2" (UID: "d4f2b753-468e-4702-8737-4e88df43eec2"). InnerVolumeSpecName "kube-api-access-5lzhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:40.047265 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.047230    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-credential-socket\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:40.047265 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.047258    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-istio-token\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:40.047265 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.047267    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-istio-envoy\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:40.047494 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.047275    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lzhr\" (UniqueName: \"kubernetes.io/projected/d4f2b753-468e-4702-8737-4e88df43eec2-kube-api-access-5lzhr\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:40.047494 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.047284    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f2b753-468e-4702-8737-4e88df43eec2-workload-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:40.047494 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.047292    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f2b753-468e-4702-8737-4e88df43eec2-istiod-ca-cert\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:40.047494 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.047299    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f2b753-468e-4702-8737-4e88df43eec2-istio-podinfo\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:40.662504 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.662466    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4f2b753-468e-4702-8737-4e88df43eec2" containerID="9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd" exitCode=0
Jul 28 17:38:40.662968 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.662529    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"
Jul 28 17:38:40.662968 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.662544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc" event={"ID":"d4f2b753-468e-4702-8737-4e88df43eec2","Type":"ContainerDied","Data":"9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd"}
Jul 28 17:38:40.662968 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.662628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc" event={"ID":"d4f2b753-468e-4702-8737-4e88df43eec2","Type":"ContainerDied","Data":"0e572c5818672a5c057cb41ae379b38944204216f2218b16352808b13e09da98"}
Jul 28 17:38:40.662968 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.662659    2559 scope.go:117] "RemoveContainer" containerID="9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd"
Jul 28 17:38:40.671505 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.671487    2559 scope.go:117] "RemoveContainer" containerID="9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd"
Jul 28 17:38:40.671764 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:38:40.671745    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd\": container with ID starting with 9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd not found: ID does not exist" containerID="9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd"
Jul 28 17:38:40.671848 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.671771    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd"} err="failed to get container status \"9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd\": rpc error: code = NotFound desc = could not find container \"9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd\": container with ID starting with 9a9c05b3f9817a4b1e6cdfe6aeecfab8bf86579bd02cd34db84f82ae14d9fdfd not found: ID does not exist"
Jul 28 17:38:40.682918 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.682868    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"]
Jul 28 17:38:40.686868 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:40.686847    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-x8kkc"]
Jul 28 17:38:41.833460 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:41.833424    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f2b753-468e-4702-8737-4e88df43eec2" path="/var/lib/kubelet/pods/d4f2b753-468e-4702-8737-4e88df43eec2/volumes"
Jul 28 17:38:42.589786 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:42.589753    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:43.093446 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:43.093415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:46.514552 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:46.514518    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:48.432359 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:48.432322    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:48.532576 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:48.532530    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:48.634311 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:48.634272    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:48.967118 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:48.967088    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:49.168111 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:49.168065    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:50.478745 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:50.478704    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:50.578969 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:50.578919    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:53.425250 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:53.425218    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"]
Jul 28 17:38:53.425671 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:53.425440    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch" podUID="0fe45dab-6862-4114-8686-eb08526f2e1d" containerName="istio-proxy" containerID="cri-o://38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2" gracePeriod=30
Jul 28 17:38:53.522624 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:53.522581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:56.477115 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.477090    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:56.588870 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.588833    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpfcd\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-kube-api-access-bpfcd\") pod \"0fe45dab-6862-4114-8686-eb08526f2e1d\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") "
Jul 28 17:38:56.589029 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.588897    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-token\") pod \"0fe45dab-6862-4114-8686-eb08526f2e1d\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") "
Jul 28 17:38:56.589029 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.588925    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-data\") pod \"0fe45dab-6862-4114-8686-eb08526f2e1d\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") "
Jul 28 17:38:56.589029 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.588967    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-podinfo\") pod \"0fe45dab-6862-4114-8686-eb08526f2e1d\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") "
Jul 28 17:38:56.589133 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589092    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-envoy\") pod \"0fe45dab-6862-4114-8686-eb08526f2e1d\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") "
Jul 28 17:38:56.589177 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589149    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe45dab-6862-4114-8686-eb08526f2e1d-istiod-ca-cert\") pod \"0fe45dab-6862-4114-8686-eb08526f2e1d\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") "
Jul 28 17:38:56.589229 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589181    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-socket\") pod \"0fe45dab-6862-4114-8686-eb08526f2e1d\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") "
Jul 28 17:38:56.589229 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589212    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-credential-socket\") pod \"0fe45dab-6862-4114-8686-eb08526f2e1d\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") "
Jul 28 17:38:56.589342 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589249    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-certs\") pod \"0fe45dab-6862-4114-8686-eb08526f2e1d\" (UID: \"0fe45dab-6862-4114-8686-eb08526f2e1d\") "
Jul 28 17:38:56.589569 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589244    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-data" (OuterVolumeSpecName: "istio-data") pod "0fe45dab-6862-4114-8686-eb08526f2e1d" (UID: "0fe45dab-6862-4114-8686-eb08526f2e1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:56.589569 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589444    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0fe45dab-6862-4114-8686-eb08526f2e1d" (UID: "0fe45dab-6862-4114-8686-eb08526f2e1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:56.589569 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589552    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0fe45dab-6862-4114-8686-eb08526f2e1d" (UID: "0fe45dab-6862-4114-8686-eb08526f2e1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:56.590001 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589628    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe45dab-6862-4114-8686-eb08526f2e1d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0fe45dab-6862-4114-8686-eb08526f2e1d" (UID: "0fe45dab-6862-4114-8686-eb08526f2e1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:56.590001 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.589780    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0fe45dab-6862-4114-8686-eb08526f2e1d" (UID: "0fe45dab-6862-4114-8686-eb08526f2e1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:56.591374 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.591348    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-kube-api-access-bpfcd" (OuterVolumeSpecName: "kube-api-access-bpfcd") pod "0fe45dab-6862-4114-8686-eb08526f2e1d" (UID: "0fe45dab-6862-4114-8686-eb08526f2e1d"). InnerVolumeSpecName "kube-api-access-bpfcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:56.591374 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.591355    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0fe45dab-6862-4114-8686-eb08526f2e1d" (UID: "0fe45dab-6862-4114-8686-eb08526f2e1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:56.591669 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.591649    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0fe45dab-6862-4114-8686-eb08526f2e1d" (UID: "0fe45dab-6862-4114-8686-eb08526f2e1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 17:38:56.591669 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.591654    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0fe45dab-6862-4114-8686-eb08526f2e1d" (UID: "0fe45dab-6862-4114-8686-eb08526f2e1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:56.690198 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.690165    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-envoy\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.690198 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.690194    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe45dab-6862-4114-8686-eb08526f2e1d-istiod-ca-cert\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.690198 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.690203    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-socket\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.690423 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.690212    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-credential-socket\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.690423 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.690220    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-workload-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.690423 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.690228    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpfcd\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-kube-api-access-bpfcd\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.690423 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.690236    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-token\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.690423 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.690243    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-data\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.690423 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.690250    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe45dab-6862-4114-8686-eb08526f2e1d-istio-podinfo\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.748633 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.748597    2559 generic.go:356] "Generic (PLEG): container finished" podID="0fe45dab-6862-4114-8686-eb08526f2e1d" containerID="38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2" exitCode=0
Jul 28 17:38:56.748817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.748642    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"
Jul 28 17:38:56.748817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.748649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch" event={"ID":"0fe45dab-6862-4114-8686-eb08526f2e1d","Type":"ContainerDied","Data":"38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2"}
Jul 28 17:38:56.748817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.748700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch" event={"ID":"0fe45dab-6862-4114-8686-eb08526f2e1d","Type":"ContainerDied","Data":"760e94800da6ea03a980ada8548cae761f1bd99c76bea19aa5a0130d5b47c171"}
Jul 28 17:38:56.748817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.748729    2559 scope.go:117] "RemoveContainer" containerID="38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2"
Jul 28 17:38:56.757218 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.757200    2559 scope.go:117] "RemoveContainer" containerID="38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2"
Jul 28 17:38:56.757487 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:38:56.757467    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2\": container with ID starting with 38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2 not found: ID does not exist" containerID="38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2"
Jul 28 17:38:56.757528 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.757496    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2"} err="failed to get container status \"38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2\": rpc error: code = NotFound desc = could not find container \"38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2\": container with ID starting with 38507fd1107dcbd0d9ee7c72da2f05671cf7d2d5450a890801e22683ebbd6fe2 not found: ID does not exist"
Jul 28 17:38:56.769540 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.769513    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"]
Jul 28 17:38:56.774351 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:56.774329    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-czpch"]
Jul 28 17:38:57.833372 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:57.833342    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fe45dab-6862-4114-8686-eb08526f2e1d" path="/var/lib/kubelet/pods/0fe45dab-6862-4114-8686-eb08526f2e1d/volumes"
Jul 28 17:38:59.359238 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:59.359202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:59.973262 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:38:59.973216    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:02.517751 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.517659    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:02.758931 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.758902    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"]
Jul 28 17:39:02.759200 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.759188    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f2b753-468e-4702-8737-4e88df43eec2" containerName="istio-proxy"
Jul 28 17:39:02.759243 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.759201    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f2b753-468e-4702-8737-4e88df43eec2" containerName="istio-proxy"
Jul 28 17:39:02.759243 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.759211    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe45dab-6862-4114-8686-eb08526f2e1d" containerName="istio-proxy"
Jul 28 17:39:02.759243 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.759217    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe45dab-6862-4114-8686-eb08526f2e1d" containerName="istio-proxy"
Jul 28 17:39:02.759332 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.759270    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f2b753-468e-4702-8737-4e88df43eec2" containerName="istio-proxy"
Jul 28 17:39:02.759332 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.759276    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe45dab-6862-4114-8686-eb08526f2e1d" containerName="istio-proxy"
Jul 28 17:39:02.762241 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.762225    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:02.765466 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.765431    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-pf4wg\""
Jul 28 17:39:02.765613 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.765494    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 28 17:39:02.765613 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.765464    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 28 17:39:02.765613 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.765563    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 28 17:39:02.765613 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.765459    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-nt7w8\""
Jul 28 17:39:02.770262 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.770239    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"]
Jul 28 17:39:02.781830 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.781799    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:02.841555 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.841521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dec16a3-3896-4fc1-bd04-802ba195022d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6\" (UID: \"7dec16a3-3896-4fc1-bd04-802ba195022d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:02.841736 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.841581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7pbn\" (UniqueName: \"kubernetes.io/projected/7dec16a3-3896-4fc1-bd04-802ba195022d-kube-api-access-b7pbn\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6\" (UID: \"7dec16a3-3896-4fc1-bd04-802ba195022d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:02.942206 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.942172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dec16a3-3896-4fc1-bd04-802ba195022d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6\" (UID: \"7dec16a3-3896-4fc1-bd04-802ba195022d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:02.942353 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.942217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7pbn\" (UniqueName: \"kubernetes.io/projected/7dec16a3-3896-4fc1-bd04-802ba195022d-kube-api-access-b7pbn\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6\" (UID: \"7dec16a3-3896-4fc1-bd04-802ba195022d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:02.944763 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.944736    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dec16a3-3896-4fc1-bd04-802ba195022d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6\" (UID: \"7dec16a3-3896-4fc1-bd04-802ba195022d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:02.950220 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:02.950194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7pbn\" (UniqueName: \"kubernetes.io/projected/7dec16a3-3896-4fc1-bd04-802ba195022d-kube-api-access-b7pbn\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6\" (UID: \"7dec16a3-3896-4fc1-bd04-802ba195022d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:03.074014 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:03.073920    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:03.212037 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:03.211932    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"]
Jul 28 17:39:03.214705 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:39:03.214672    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dec16a3_3896_4fc1_bd04_802ba195022d.slice/crio-9d5ca906ee90c8e85859bed27a0b9bf0d3cc485ab04a21632f60a3280dbfdd58 WatchSource:0}: Error finding container 9d5ca906ee90c8e85859bed27a0b9bf0d3cc485ab04a21632f60a3280dbfdd58: Status 404 returned error can't find the container with id 9d5ca906ee90c8e85859bed27a0b9bf0d3cc485ab04a21632f60a3280dbfdd58
Jul 28 17:39:03.783083 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:03.783045    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6" event={"ID":"7dec16a3-3896-4fc1-bd04-802ba195022d","Type":"ContainerStarted","Data":"5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79"}
Jul 28 17:39:03.783711 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:03.783105    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6" event={"ID":"7dec16a3-3896-4fc1-bd04-802ba195022d","Type":"ContainerStarted","Data":"9d5ca906ee90c8e85859bed27a0b9bf0d3cc485ab04a21632f60a3280dbfdd58"}
Jul 28 17:39:03.783711 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:03.783147    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:03.801289 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:03.801238    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6" podStartSLOduration=1.8012250189999999 podStartE2EDuration="1.801225019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:03.799537339 +0000 UTC m=+984.454032317" watchObservedRunningTime="2026-07-28 17:39:03.801225019 +0000 UTC m=+984.455719994"
Jul 28 17:39:04.920192 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:04.920146    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:06.233523 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:06.233489    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:15.117676 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:15.117633    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:15.196295 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:15.196244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:21.064485 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:21.064448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:23.381999 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:23.381965    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:23.468513 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:23.468482    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:29.960568 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:29.960517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:34.788764 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:34.788731    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:39:36.224392 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:36.224351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:36.331331 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:39:36.331294    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:00.567165 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:00.567125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:00.760059 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:00.760002    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:00.991165 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:00.991125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:01.191942 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.191900    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"]
Jul 28 17:40:01.196586 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.196560    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:01.200699 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.200672    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-m4fbq\""
Jul 28 17:40:01.200901 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.200766    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 17:40:01.200901 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.200817    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-pbmfm\""
Jul 28 17:40:01.200901 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.200818    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 17:40:01.201051 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.200963    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 17:40:01.216167 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.216142    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"]
Jul 28 17:40:01.225446 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.225414    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:01.320731 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.320622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zxv\" (UniqueName: \"kubernetes.io/projected/8b06ca06-330d-41a7-b48d-0f889c510ce8-kube-api-access-h8zxv\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48\" (UID: \"8b06ca06-330d-41a7-b48d-0f889c510ce8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:01.320731 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.320686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b06ca06-330d-41a7-b48d-0f889c510ce8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48\" (UID: \"8b06ca06-330d-41a7-b48d-0f889c510ce8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:01.421696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.421663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b06ca06-330d-41a7-b48d-0f889c510ce8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48\" (UID: \"8b06ca06-330d-41a7-b48d-0f889c510ce8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:01.421853 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.421712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zxv\" (UniqueName: \"kubernetes.io/projected/8b06ca06-330d-41a7-b48d-0f889c510ce8-kube-api-access-h8zxv\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48\" (UID: \"8b06ca06-330d-41a7-b48d-0f889c510ce8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:01.424361 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.424334    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b06ca06-330d-41a7-b48d-0f889c510ce8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48\" (UID: \"8b06ca06-330d-41a7-b48d-0f889c510ce8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:01.431134 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.431110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zxv\" (UniqueName: \"kubernetes.io/projected/8b06ca06-330d-41a7-b48d-0f889c510ce8-kube-api-access-h8zxv\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48\" (UID: \"8b06ca06-330d-41a7-b48d-0f889c510ce8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:01.506997 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.506968    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:01.643147 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:01.642992    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"]
Jul 28 17:40:01.645698 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:40:01.645666    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b06ca06_330d_41a7_b48d_0f889c510ce8.slice/crio-9122f85304231132c92b636d0f439dee99ba5926427b3092b8a762d527c46bbe WatchSource:0}: Error finding container 9122f85304231132c92b636d0f439dee99ba5926427b3092b8a762d527c46bbe: Status 404 returned error can't find the container with id 9122f85304231132c92b636d0f439dee99ba5926427b3092b8a762d527c46bbe
Jul 28 17:40:02.026009 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:02.025977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48" event={"ID":"8b06ca06-330d-41a7-b48d-0f889c510ce8","Type":"ContainerStarted","Data":"c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79"}
Jul 28 17:40:02.026195 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:02.026031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48" event={"ID":"8b06ca06-330d-41a7-b48d-0f889c510ce8","Type":"ContainerStarted","Data":"9122f85304231132c92b636d0f439dee99ba5926427b3092b8a762d527c46bbe"}
Jul 28 17:40:02.026195 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:02.026060    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:02.044124 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:02.044074    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48" podStartSLOduration=1.044053344 podStartE2EDuration="1.044053344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:02.043175171 +0000 UTC m=+1042.697670146" watchObservedRunningTime="2026-07-28 17:40:02.044053344 +0000 UTC m=+1042.698548320"
Jul 28 17:40:04.178462 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:04.178425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:04.278513 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:04.278477    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:14.461012 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:14.460976    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:33.033219 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:33.033190    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:40:39.349231 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:40:39.349189    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:06.182293 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:06.182200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:06.283093 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:06.283053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:22.879481 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:22.879438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:37.783919 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:37.783865    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"]
Jul 28 17:42:37.785384 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:37.785350    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:37.849576 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:37.849521    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:43.275467 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:43.275434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:43.313469 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:43.313433    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"]
Jul 28 17:42:43.392670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:43.392639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:44.789510 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:44.789463    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6" podUID="7dec16a3-3896-4fc1-bd04-802ba195022d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:42:52.803830 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:52.803766    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6" podUID="7dec16a3-3896-4fc1-bd04-802ba195022d" containerName="main" containerID="cri-o://5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79" gracePeriod=45
Jul 28 17:42:53.032606 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.032569    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48" podUID="8b06ca06-330d-41a7-b48d-0f889c510ce8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:42:53.059293 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.059240    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:42:53.127105 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.127070    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7pbn\" (UniqueName: \"kubernetes.io/projected/7dec16a3-3896-4fc1-bd04-802ba195022d-kube-api-access-b7pbn\") pod \"7dec16a3-3896-4fc1-bd04-802ba195022d\" (UID: \"7dec16a3-3896-4fc1-bd04-802ba195022d\") "
Jul 28 17:42:53.127275 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.127131    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dec16a3-3896-4fc1-bd04-802ba195022d-tls-certs\") pod \"7dec16a3-3896-4fc1-bd04-802ba195022d\" (UID: \"7dec16a3-3896-4fc1-bd04-802ba195022d\") "
Jul 28 17:42:53.129444 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.129408    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dec16a3-3896-4fc1-bd04-802ba195022d-kube-api-access-b7pbn" (OuterVolumeSpecName: "kube-api-access-b7pbn") pod "7dec16a3-3896-4fc1-bd04-802ba195022d" (UID: "7dec16a3-3896-4fc1-bd04-802ba195022d"). InnerVolumeSpecName "kube-api-access-b7pbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:53.129444 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.129408    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dec16a3-3896-4fc1-bd04-802ba195022d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7dec16a3-3896-4fc1-bd04-802ba195022d" (UID: "7dec16a3-3896-4fc1-bd04-802ba195022d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:53.227838 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.227800    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dec16a3-3896-4fc1-bd04-802ba195022d-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:42:53.227838 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.227830    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7pbn\" (UniqueName: \"kubernetes.io/projected/7dec16a3-3896-4fc1-bd04-802ba195022d-kube-api-access-b7pbn\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:42:53.745217 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.745185    2559 generic.go:356] "Generic (PLEG): container finished" podID="7dec16a3-3896-4fc1-bd04-802ba195022d" containerID="5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79" exitCode=0
Jul 28 17:42:53.745410 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.745239    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"
Jul 28 17:42:53.745410 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.745248    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6" event={"ID":"7dec16a3-3896-4fc1-bd04-802ba195022d","Type":"ContainerDied","Data":"5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79"}
Jul 28 17:42:53.745410 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.745291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6" event={"ID":"7dec16a3-3896-4fc1-bd04-802ba195022d","Type":"ContainerDied","Data":"9d5ca906ee90c8e85859bed27a0b9bf0d3cc485ab04a21632f60a3280dbfdd58"}
Jul 28 17:42:53.745410 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.745312    2559 scope.go:117] "RemoveContainer" containerID="5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79"
Jul 28 17:42:53.754427 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.754411    2559 scope.go:117] "RemoveContainer" containerID="5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79"
Jul 28 17:42:53.754690 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:42:53.754672    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79\": container with ID starting with 5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79 not found: ID does not exist" containerID="5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79"
Jul 28 17:42:53.754735 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.754699    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79"} err="failed to get container status \"5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79\": rpc error: code = NotFound desc = could not find container \"5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79\": container with ID starting with 5eed2de949cfc38194d7dc8dcc298a8771fb5965af110b8fed6ae6f0d2853b79 not found: ID does not exist"
Jul 28 17:42:53.767223 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.767180    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"]
Jul 28 17:42:53.769006 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.768983    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-bkvb6"]
Jul 28 17:42:53.832575 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:53.832548    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dec16a3-3896-4fc1-bd04-802ba195022d" path="/var/lib/kubelet/pods/7dec16a3-3896-4fc1-bd04-802ba195022d/volumes"
Jul 28 17:42:57.799381 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:57.799349    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:57.950186 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:57.950147    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:58.332800 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.332751    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48" podUID="8b06ca06-330d-41a7-b48d-0f889c510ce8" containerName="main" containerID="cri-o://c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79" gracePeriod=45
Jul 28 17:42:58.584665 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.584601    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:42:58.670714 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.670677    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b06ca06-330d-41a7-b48d-0f889c510ce8-tls-certs\") pod \"8b06ca06-330d-41a7-b48d-0f889c510ce8\" (UID: \"8b06ca06-330d-41a7-b48d-0f889c510ce8\") "
Jul 28 17:42:58.670912 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.670722    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8zxv\" (UniqueName: \"kubernetes.io/projected/8b06ca06-330d-41a7-b48d-0f889c510ce8-kube-api-access-h8zxv\") pod \"8b06ca06-330d-41a7-b48d-0f889c510ce8\" (UID: \"8b06ca06-330d-41a7-b48d-0f889c510ce8\") "
Jul 28 17:42:58.673205 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.673173    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b06ca06-330d-41a7-b48d-0f889c510ce8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b06ca06-330d-41a7-b48d-0f889c510ce8" (UID: "8b06ca06-330d-41a7-b48d-0f889c510ce8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:58.673205 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.673178    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b06ca06-330d-41a7-b48d-0f889c510ce8-kube-api-access-h8zxv" (OuterVolumeSpecName: "kube-api-access-h8zxv") pod "8b06ca06-330d-41a7-b48d-0f889c510ce8" (UID: "8b06ca06-330d-41a7-b48d-0f889c510ce8"). InnerVolumeSpecName "kube-api-access-h8zxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:58.767573 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.767539    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b06ca06-330d-41a7-b48d-0f889c510ce8" containerID="c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79" exitCode=0
Jul 28 17:42:58.767748 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.767582    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"
Jul 28 17:42:58.767748 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.767616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48" event={"ID":"8b06ca06-330d-41a7-b48d-0f889c510ce8","Type":"ContainerDied","Data":"c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79"}
Jul 28 17:42:58.767748 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.767690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48" event={"ID":"8b06ca06-330d-41a7-b48d-0f889c510ce8","Type":"ContainerDied","Data":"9122f85304231132c92b636d0f439dee99ba5926427b3092b8a762d527c46bbe"}
Jul 28 17:42:58.767748 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.767713    2559 scope.go:117] "RemoveContainer" containerID="c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79"
Jul 28 17:42:58.771267 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.771247    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b06ca06-330d-41a7-b48d-0f889c510ce8-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:42:58.771267 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.771267    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8zxv\" (UniqueName: \"kubernetes.io/projected/8b06ca06-330d-41a7-b48d-0f889c510ce8-kube-api-access-h8zxv\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:42:58.776484 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.776470    2559 scope.go:117] "RemoveContainer" containerID="c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79"
Jul 28 17:42:58.776747 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:42:58.776729    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79\": container with ID starting with c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79 not found: ID does not exist" containerID="c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79"
Jul 28 17:42:58.776796 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.776755    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79"} err="failed to get container status \"c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79\": rpc error: code = NotFound desc = could not find container \"c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79\": container with ID starting with c7227d95c3a6377334a9512e602c0589d46272531e13548f6e9a12384db69a79 not found: ID does not exist"
Jul 28 17:42:58.788982 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.788955    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"]
Jul 28 17:42:58.792031 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:58.792011    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schewfs48"]
Jul 28 17:42:59.832695 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:42:59.832663    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b06ca06-330d-41a7-b48d-0f889c510ce8" path="/var/lib/kubelet/pods/8b06ca06-330d-41a7-b48d-0f889c510ce8/volumes"
Jul 28 17:43:01.124220 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.124166    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.368727 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.368696    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"]
Jul 28 17:43:01.369024 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.369011    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dec16a3-3896-4fc1-bd04-802ba195022d" containerName="main"
Jul 28 17:43:01.369067 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.369026    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dec16a3-3896-4fc1-bd04-802ba195022d" containerName="main"
Jul 28 17:43:01.369067 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.369041    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b06ca06-330d-41a7-b48d-0f889c510ce8" containerName="main"
Jul 28 17:43:01.369067 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.369047    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b06ca06-330d-41a7-b48d-0f889c510ce8" containerName="main"
Jul 28 17:43:01.369157 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.369092    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dec16a3-3896-4fc1-bd04-802ba195022d" containerName="main"
Jul 28 17:43:01.369157 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.369099    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b06ca06-330d-41a7-b48d-0f889c510ce8" containerName="main"
Jul 28 17:43:01.373444 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.373424    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:01.375935 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.375849    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 28 17:43:01.376028 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.375850    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-ppgf4\""
Jul 28 17:43:01.376977 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.376945    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 28 17:43:01.376977 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.376969    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 28 17:43:01.377129 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.377018    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-l5z6k\""
Jul 28 17:43:01.385792 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.385771    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"]
Jul 28 17:43:01.429012 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.428978    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.496062 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.496017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd524c-e28c-457f-9887-6ebd0c009b05-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4\" (UID: \"cecd524c-e28c-457f-9887-6ebd0c009b05\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:01.496062 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.496067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59drr\" (UniqueName: \"kubernetes.io/projected/cecd524c-e28c-457f-9887-6ebd0c009b05-kube-api-access-59drr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4\" (UID: \"cecd524c-e28c-457f-9887-6ebd0c009b05\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:01.596564 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.596532    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd524c-e28c-457f-9887-6ebd0c009b05-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4\" (UID: \"cecd524c-e28c-457f-9887-6ebd0c009b05\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:01.596762 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.596573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59drr\" (UniqueName: \"kubernetes.io/projected/cecd524c-e28c-457f-9887-6ebd0c009b05-kube-api-access-59drr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4\" (UID: \"cecd524c-e28c-457f-9887-6ebd0c009b05\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:01.599213 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.599188    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd524c-e28c-457f-9887-6ebd0c009b05-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4\" (UID: \"cecd524c-e28c-457f-9887-6ebd0c009b05\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:01.607410 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.607378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59drr\" (UniqueName: \"kubernetes.io/projected/cecd524c-e28c-457f-9887-6ebd0c009b05-kube-api-access-59drr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4\" (UID: \"cecd524c-e28c-457f-9887-6ebd0c009b05\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:01.683443 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.683341    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:01.821453 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:01.821392    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"]
Jul 28 17:43:01.823835 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:43:01.823801    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecd524c_e28c_457f_9887_6ebd0c009b05.slice/crio-aadbeffddef76c88b25e6af3937cd9b523460bf493d1627df044a4d70a81a9a4 WatchSource:0}: Error finding container aadbeffddef76c88b25e6af3937cd9b523460bf493d1627df044a4d70a81a9a4: Status 404 returned error can't find the container with id aadbeffddef76c88b25e6af3937cd9b523460bf493d1627df044a4d70a81a9a4
Jul 28 17:43:02.789641 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:02.789608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4" event={"ID":"cecd524c-e28c-457f-9887-6ebd0c009b05","Type":"ContainerStarted","Data":"41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a"}
Jul 28 17:43:02.790134 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:02.789658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4" event={"ID":"cecd524c-e28c-457f-9887-6ebd0c009b05","Type":"ContainerStarted","Data":"aadbeffddef76c88b25e6af3937cd9b523460bf493d1627df044a4d70a81a9a4"}
Jul 28 17:43:02.790134 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:02.789713    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:02.816007 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:02.815948    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4" podStartSLOduration=1.815931622 podStartE2EDuration="1.815931622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:02.815136308 +0000 UTC m=+1223.469631283" watchObservedRunningTime="2026-07-28 17:43:02.815931622 +0000 UTC m=+1223.470426598"
Jul 28 17:43:04.605807 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:04.605769    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:10.647710 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:10.647668    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:10.773386 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:10.773338    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:18.467272 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:18.467233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:18.565140 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:18.565104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:21.670474 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:21.670437    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:25.613590 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:25.613551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:31.564923 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.564812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:31.808454 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.808417    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"]
Jul 28 17:43:31.811694 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.811675    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:43:31.816240 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.816186    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-fnftk\""
Jul 28 17:43:31.816850 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.816818    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-cnf5p\""
Jul 28 17:43:31.816994 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.816853    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 28 17:43:31.816994 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.816931    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 28 17:43:31.816994 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.816970    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 28 17:43:31.833038 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.833011    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"]
Jul 28 17:43:31.843321 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.843291    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9c8415-82bb-44a6-83cd-0dd42c961263-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc\" (UID: \"2e9c8415-82bb-44a6-83cd-0dd42c961263\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:43:31.843442 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.843400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dd9p\" (UniqueName: \"kubernetes.io/projected/2e9c8415-82bb-44a6-83cd-0dd42c961263-kube-api-access-2dd9p\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc\" (UID: \"2e9c8415-82bb-44a6-83cd-0dd42c961263\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:43:31.855534 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.855504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:31.944658 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.944615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dd9p\" (UniqueName: \"kubernetes.io/projected/2e9c8415-82bb-44a6-83cd-0dd42c961263-kube-api-access-2dd9p\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc\" (UID: \"2e9c8415-82bb-44a6-83cd-0dd42c961263\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:43:31.944844 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.944691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9c8415-82bb-44a6-83cd-0dd42c961263-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc\" (UID: \"2e9c8415-82bb-44a6-83cd-0dd42c961263\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:43:31.947321 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.947294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9c8415-82bb-44a6-83cd-0dd42c961263-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc\" (UID: \"2e9c8415-82bb-44a6-83cd-0dd42c961263\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:43:31.960958 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:31.960930    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dd9p\" (UniqueName: \"kubernetes.io/projected/2e9c8415-82bb-44a6-83cd-0dd42c961263-kube-api-access-2dd9p\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc\" (UID: \"2e9c8415-82bb-44a6-83cd-0dd42c961263\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:43:32.121955 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:32.121923    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:43:32.257134 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:32.257110    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"]
Jul 28 17:43:32.259913 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:43:32.259865    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9c8415_82bb_44a6_83cd_0dd42c961263.slice/crio-5cc753c6bacc8aa4b677d144f5978927c19f6882a80ddf6548a39306811a9589 WatchSource:0}: Error finding container 5cc753c6bacc8aa4b677d144f5978927c19f6882a80ddf6548a39306811a9589: Status 404 returned error can't find the container with id 5cc753c6bacc8aa4b677d144f5978927c19f6882a80ddf6548a39306811a9589
Jul 28 17:43:32.506992 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:32.506900    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:32.585931 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:32.585896    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:32.916669 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:32.916636    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc" event={"ID":"2e9c8415-82bb-44a6-83cd-0dd42c961263","Type":"ContainerStarted","Data":"0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287"}
Jul 28 17:43:32.916913 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:32.916687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc" event={"ID":"2e9c8415-82bb-44a6-83cd-0dd42c961263","Type":"ContainerStarted","Data":"5cc753c6bacc8aa4b677d144f5978927c19f6882a80ddf6548a39306811a9589"}
Jul 28 17:43:32.916913 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:32.916718    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:43:32.935413 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:32.935363    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc" podStartSLOduration=1.935349511 podStartE2EDuration="1.935349511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:32.934439316 +0000 UTC m=+1253.588934290" watchObservedRunningTime="2026-07-28 17:43:32.935349511 +0000 UTC m=+1253.589844490"
Jul 28 17:43:33.797041 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:33.797013    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:43:41.088571 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:43:41.088528    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:03.923108 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:03.923080    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:44:05.262890 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:05.262830    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:05.271372 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:05.271340    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"]
Jul 28 17:44:05.404172 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:05.404127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:13.923614 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:13.923577    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc" podUID="2e9c8415-82bb-44a6-83cd-0dd42c961263" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:44:20.291405 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:20.291366    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc" podUID="2e9c8415-82bb-44a6-83cd-0dd42c961263" containerName="main" containerID="cri-o://0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287" gracePeriod=45
Jul 28 17:44:20.540112 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:20.540083    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:44:20.637607 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:20.637574    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9c8415-82bb-44a6-83cd-0dd42c961263-tls-certs\") pod \"2e9c8415-82bb-44a6-83cd-0dd42c961263\" (UID: \"2e9c8415-82bb-44a6-83cd-0dd42c961263\") "
Jul 28 17:44:20.637760 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:20.637634    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dd9p\" (UniqueName: \"kubernetes.io/projected/2e9c8415-82bb-44a6-83cd-0dd42c961263-kube-api-access-2dd9p\") pod \"2e9c8415-82bb-44a6-83cd-0dd42c961263\" (UID: \"2e9c8415-82bb-44a6-83cd-0dd42c961263\") "
Jul 28 17:44:20.639906 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:20.639859    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9c8415-82bb-44a6-83cd-0dd42c961263-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e9c8415-82bb-44a6-83cd-0dd42c961263" (UID: "2e9c8415-82bb-44a6-83cd-0dd42c961263"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:44:20.640021 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:20.639921    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9c8415-82bb-44a6-83cd-0dd42c961263-kube-api-access-2dd9p" (OuterVolumeSpecName: "kube-api-access-2dd9p") pod "2e9c8415-82bb-44a6-83cd-0dd42c961263" (UID: "2e9c8415-82bb-44a6-83cd-0dd42c961263"). InnerVolumeSpecName "kube-api-access-2dd9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:44:20.739082 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:20.739032    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dd9p\" (UniqueName: \"kubernetes.io/projected/2e9c8415-82bb-44a6-83cd-0dd42c961263-kube-api-access-2dd9p\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:44:20.739082 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:20.739073    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9c8415-82bb-44a6-83cd-0dd42c961263-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:44:21.127353 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.127305    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e9c8415-82bb-44a6-83cd-0dd42c961263" containerID="0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287" exitCode=0
Jul 28 17:44:21.127531 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.127361    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"
Jul 28 17:44:21.127531 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.127372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc" event={"ID":"2e9c8415-82bb-44a6-83cd-0dd42c961263","Type":"ContainerDied","Data":"0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287"}
Jul 28 17:44:21.127531 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.127416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc" event={"ID":"2e9c8415-82bb-44a6-83cd-0dd42c961263","Type":"ContainerDied","Data":"5cc753c6bacc8aa4b677d144f5978927c19f6882a80ddf6548a39306811a9589"}
Jul 28 17:44:21.127531 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.127437    2559 scope.go:117] "RemoveContainer" containerID="0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287"
Jul 28 17:44:21.136018 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.136000    2559 scope.go:117] "RemoveContainer" containerID="0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287"
Jul 28 17:44:21.136284 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:44:21.136266    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287\": container with ID starting with 0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287 not found: ID does not exist" containerID="0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287"
Jul 28 17:44:21.136328 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.136295    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287"} err="failed to get container status \"0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287\": rpc error: code = NotFound desc = could not find container \"0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287\": container with ID starting with 0a818e023c6ce885f18a05f77c49c847f62007909f4906d9d61b09cf6d3ca287 not found: ID does not exist"
Jul 28 17:44:21.151112 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.151087    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"]
Jul 28 17:44:21.158520 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.158496    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5fkmhc"]
Jul 28 17:44:21.832717 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:21.832685    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e9c8415-82bb-44a6-83cd-0dd42c961263" path="/var/lib/kubelet/pods/2e9c8415-82bb-44a6-83cd-0dd42c961263/volumes"
Jul 28 17:44:25.012092 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:25.012061    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:25.186387 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:25.186345    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:32.022442 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:32.022402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:38.848793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:38.848750    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:38.949617 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:38.949574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:48.429589 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.429553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:48.606920 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.606869    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"]
Jul 28 17:44:48.607261 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.607244    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9c8415-82bb-44a6-83cd-0dd42c961263" containerName="main"
Jul 28 17:44:48.607350 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.607266    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9c8415-82bb-44a6-83cd-0dd42c961263" containerName="main"
Jul 28 17:44:48.607407 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.607363    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9c8415-82bb-44a6-83cd-0dd42c961263" containerName="main"
Jul 28 17:44:48.611196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.611174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:44:48.613890 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.613860    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 28 17:44:48.614713 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.614690    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-ckfkd\""
Jul 28 17:44:48.614713 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.614704    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 28 17:44:48.614865 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.614740    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-wmwxb\""
Jul 28 17:44:48.614976 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.614959    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 28 17:44:48.628855 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.628832    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"]
Jul 28 17:44:48.647144 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.646950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:48.669978 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.669937    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7nv5\" (UniqueName: \"kubernetes.io/projected/13902b3c-5a03-4a81-b62a-232f6798a58b-kube-api-access-j7nv5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl\" (UID: \"13902b3c-5a03-4a81-b62a-232f6798a58b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:44:48.670108 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.669983    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13902b3c-5a03-4a81-b62a-232f6798a58b-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl\" (UID: \"13902b3c-5a03-4a81-b62a-232f6798a58b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:44:48.771517 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.771419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7nv5\" (UniqueName: \"kubernetes.io/projected/13902b3c-5a03-4a81-b62a-232f6798a58b-kube-api-access-j7nv5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl\" (UID: \"13902b3c-5a03-4a81-b62a-232f6798a58b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:44:48.771517 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.771471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13902b3c-5a03-4a81-b62a-232f6798a58b-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl\" (UID: \"13902b3c-5a03-4a81-b62a-232f6798a58b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:44:48.774234 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.774210    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13902b3c-5a03-4a81-b62a-232f6798a58b-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl\" (UID: \"13902b3c-5a03-4a81-b62a-232f6798a58b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:44:48.780376 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.780345    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7nv5\" (UniqueName: \"kubernetes.io/projected/13902b3c-5a03-4a81-b62a-232f6798a58b-kube-api-access-j7nv5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl\" (UID: \"13902b3c-5a03-4a81-b62a-232f6798a58b\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:44:48.921408 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:48.921361    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:44:49.059364 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:49.059336    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"]
Jul 28 17:44:49.062088 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:44:49.062049    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13902b3c_5a03_4a81_b62a_232f6798a58b.slice/crio-34019ec63a807cd29e16b5be9395456cc678587109079ff56de6055df38cc9ac WatchSource:0}: Error finding container 34019ec63a807cd29e16b5be9395456cc678587109079ff56de6055df38cc9ac: Status 404 returned error can't find the container with id 34019ec63a807cd29e16b5be9395456cc678587109079ff56de6055df38cc9ac
Jul 28 17:44:49.243186 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:49.243148    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl" event={"ID":"13902b3c-5a03-4a81-b62a-232f6798a58b","Type":"ContainerStarted","Data":"89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1"}
Jul 28 17:44:49.243186 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:49.243204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl" event={"ID":"13902b3c-5a03-4a81-b62a-232f6798a58b","Type":"ContainerStarted","Data":"34019ec63a807cd29e16b5be9395456cc678587109079ff56de6055df38cc9ac"}
Jul 28 17:44:49.243457 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:49.243224    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:44:49.265138 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:44:49.265091    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl" podStartSLOduration=1.265077138 podStartE2EDuration="1.265077138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:49.263440868 +0000 UTC m=+1329.917935844" watchObservedRunningTime="2026-07-28 17:44:49.265077138 +0000 UTC m=+1329.919572112"
Jul 28 17:45:02.086234 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:02.086143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:12.764641 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:12.764601    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"]
Jul 28 17:45:12.785859 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:12.785828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:12.883952 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:12.883837    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:13.797022 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:13.796976    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4" podUID="cecd524c-e28c-457f-9887-6ebd0c009b05" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:45:20.249138 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:20.249110    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:45:21.534503 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:21.534464    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"]
Jul 28 17:45:21.540024 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:21.539991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:21.615464 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:21.615427    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:23.796460 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:23.796428    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4" podUID="cecd524c-e28c-457f-9887-6ebd0c009b05" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:45:27.785687 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:27.785627    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4" podUID="cecd524c-e28c-457f-9887-6ebd0c009b05" containerName="main" containerID="cri-o://41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a" gracePeriod=45
Jul 28 17:45:28.036084 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.036028    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:45:28.092197 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.092161    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59drr\" (UniqueName: \"kubernetes.io/projected/cecd524c-e28c-457f-9887-6ebd0c009b05-kube-api-access-59drr\") pod \"cecd524c-e28c-457f-9887-6ebd0c009b05\" (UID: \"cecd524c-e28c-457f-9887-6ebd0c009b05\") "
Jul 28 17:45:28.094346 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.094317    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecd524c-e28c-457f-9887-6ebd0c009b05-kube-api-access-59drr" (OuterVolumeSpecName: "kube-api-access-59drr") pod "cecd524c-e28c-457f-9887-6ebd0c009b05" (UID: "cecd524c-e28c-457f-9887-6ebd0c009b05"). InnerVolumeSpecName "kube-api-access-59drr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:28.192569 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.192525    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd524c-e28c-457f-9887-6ebd0c009b05-tls-certs\") pod \"cecd524c-e28c-457f-9887-6ebd0c009b05\" (UID: \"cecd524c-e28c-457f-9887-6ebd0c009b05\") "
Jul 28 17:45:28.192770 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.192749    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59drr\" (UniqueName: \"kubernetes.io/projected/cecd524c-e28c-457f-9887-6ebd0c009b05-kube-api-access-59drr\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.194678 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.194654    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cecd524c-e28c-457f-9887-6ebd0c009b05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cecd524c-e28c-457f-9887-6ebd0c009b05" (UID: "cecd524c-e28c-457f-9887-6ebd0c009b05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:28.293235 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.293140    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd524c-e28c-457f-9887-6ebd0c009b05-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.411544 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.411507    2559 generic.go:356] "Generic (PLEG): container finished" podID="cecd524c-e28c-457f-9887-6ebd0c009b05" containerID="41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a" exitCode=0
Jul 28 17:45:28.411740 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.411567    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"
Jul 28 17:45:28.411740 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.411585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4" event={"ID":"cecd524c-e28c-457f-9887-6ebd0c009b05","Type":"ContainerDied","Data":"41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a"}
Jul 28 17:45:28.411740 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.411665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4" event={"ID":"cecd524c-e28c-457f-9887-6ebd0c009b05","Type":"ContainerDied","Data":"aadbeffddef76c88b25e6af3937cd9b523460bf493d1627df044a4d70a81a9a4"}
Jul 28 17:45:28.411740 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.411696    2559 scope.go:117] "RemoveContainer" containerID="41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a"
Jul 28 17:45:28.420191 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.420173    2559 scope.go:117] "RemoveContainer" containerID="41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a"
Jul 28 17:45:28.420447 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:45:28.420424    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a\": container with ID starting with 41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a not found: ID does not exist" containerID="41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a"
Jul 28 17:45:28.420551 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.420460    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a"} err="failed to get container status \"41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a\": rpc error: code = NotFound desc = could not find container \"41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a\": container with ID starting with 41196bf1c5002bd72a29a976b2816f6ce01d348b8a0434a494e16296758efb6a not found: ID does not exist"
Jul 28 17:45:28.431852 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.431819    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"]
Jul 28 17:45:28.435581 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:28.435555    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dczmmm4"]
Jul 28 17:45:29.833202 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:29.833161    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cecd524c-e28c-457f-9887-6ebd0c009b05" path="/var/lib/kubelet/pods/cecd524c-e28c-457f-9887-6ebd0c009b05/volumes"
Jul 28 17:45:30.249295 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:30.249256    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl" podUID="13902b3c-5a03-4a81-b62a-232f6798a58b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:45:32.906101 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:32.906062    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:33.008481 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:33.008447    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:35.981305 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:35.981275    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:36.278349 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.278276    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"]
Jul 28 17:45:36.278573 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.278561    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cecd524c-e28c-457f-9887-6ebd0c009b05" containerName="main"
Jul 28 17:45:36.278614 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.278574    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cecd524c-e28c-457f-9887-6ebd0c009b05" containerName="main"
Jul 28 17:45:36.278655 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.278637    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cecd524c-e28c-457f-9887-6ebd0c009b05" containerName="main"
Jul 28 17:45:36.283049 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.283031    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:45:36.286473 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.286437    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 28 17:45:36.287215 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.287196    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 28 17:45:36.287323 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.287211    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 28 17:45:36.287323 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.287248    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-qxx6g\""
Jul 28 17:45:36.287478 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.287380    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-nd8p6\""
Jul 28 17:45:36.299495 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.299472    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"]
Jul 28 17:45:36.320944 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.320914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:36.349161 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.349124    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2337a18f-c119-4af1-8c17-082272711207-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf\" (UID: \"2337a18f-c119-4af1-8c17-082272711207\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:45:36.349282 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.349230    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcmw\" (UniqueName: \"kubernetes.io/projected/2337a18f-c119-4af1-8c17-082272711207-kube-api-access-xfcmw\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf\" (UID: \"2337a18f-c119-4af1-8c17-082272711207\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:45:36.449995 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.449964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2337a18f-c119-4af1-8c17-082272711207-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf\" (UID: \"2337a18f-c119-4af1-8c17-082272711207\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:45:36.450165 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.450061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcmw\" (UniqueName: \"kubernetes.io/projected/2337a18f-c119-4af1-8c17-082272711207-kube-api-access-xfcmw\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf\" (UID: \"2337a18f-c119-4af1-8c17-082272711207\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:45:36.452705 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.452683    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2337a18f-c119-4af1-8c17-082272711207-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf\" (UID: \"2337a18f-c119-4af1-8c17-082272711207\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:45:36.460135 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.460107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcmw\" (UniqueName: \"kubernetes.io/projected/2337a18f-c119-4af1-8c17-082272711207-kube-api-access-xfcmw\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf\" (UID: \"2337a18f-c119-4af1-8c17-082272711207\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:45:36.563336 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.563238    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl" podUID="13902b3c-5a03-4a81-b62a-232f6798a58b" containerName="main" containerID="cri-o://89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1" gracePeriod=45
Jul 28 17:45:36.592863 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.592839    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:45:36.731419 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.731393    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"]
Jul 28 17:45:36.818362 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.818303    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:45:36.853522 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.853495    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13902b3c-5a03-4a81-b62a-232f6798a58b-tls-certs\") pod \"13902b3c-5a03-4a81-b62a-232f6798a58b\" (UID: \"13902b3c-5a03-4a81-b62a-232f6798a58b\") "
Jul 28 17:45:36.853699 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.853550    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7nv5\" (UniqueName: \"kubernetes.io/projected/13902b3c-5a03-4a81-b62a-232f6798a58b-kube-api-access-j7nv5\") pod \"13902b3c-5a03-4a81-b62a-232f6798a58b\" (UID: \"13902b3c-5a03-4a81-b62a-232f6798a58b\") "
Jul 28 17:45:36.856059 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.856029    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13902b3c-5a03-4a81-b62a-232f6798a58b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13902b3c-5a03-4a81-b62a-232f6798a58b" (UID: "13902b3c-5a03-4a81-b62a-232f6798a58b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:36.856210 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.856066    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13902b3c-5a03-4a81-b62a-232f6798a58b-kube-api-access-j7nv5" (OuterVolumeSpecName: "kube-api-access-j7nv5") pod "13902b3c-5a03-4a81-b62a-232f6798a58b" (UID: "13902b3c-5a03-4a81-b62a-232f6798a58b"). InnerVolumeSpecName "kube-api-access-j7nv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:36.954292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.954257    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13902b3c-5a03-4a81-b62a-232f6798a58b-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:45:36.954292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:36.954284    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7nv5\" (UniqueName: \"kubernetes.io/projected/13902b3c-5a03-4a81-b62a-232f6798a58b-kube-api-access-j7nv5\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:45:37.451378 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.451348    2559 generic.go:356] "Generic (PLEG): container finished" podID="13902b3c-5a03-4a81-b62a-232f6798a58b" containerID="89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1" exitCode=0
Jul 28 17:45:37.451862 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.451395    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"
Jul 28 17:45:37.451862 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.451418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl" event={"ID":"13902b3c-5a03-4a81-b62a-232f6798a58b","Type":"ContainerDied","Data":"89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1"}
Jul 28 17:45:37.451862 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.451460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl" event={"ID":"13902b3c-5a03-4a81-b62a-232f6798a58b","Type":"ContainerDied","Data":"34019ec63a807cd29e16b5be9395456cc678587109079ff56de6055df38cc9ac"}
Jul 28 17:45:37.451862 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.451483    2559 scope.go:117] "RemoveContainer" containerID="89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1"
Jul 28 17:45:37.453407 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.453379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf" event={"ID":"2337a18f-c119-4af1-8c17-082272711207","Type":"ContainerStarted","Data":"0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99"}
Jul 28 17:45:37.453534 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.453424    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf" event={"ID":"2337a18f-c119-4af1-8c17-082272711207","Type":"ContainerStarted","Data":"dcbd395bc094af55d4b5d0e3c35c05e710162558f719d338e1d7d16fed3e597b"}
Jul 28 17:45:37.453534 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.453468    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:45:37.467025 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.467007    2559 scope.go:117] "RemoveContainer" containerID="89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1"
Jul 28 17:45:37.467292 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:45:37.467273    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1\": container with ID starting with 89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1 not found: ID does not exist" containerID="89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1"
Jul 28 17:45:37.467356 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.467301    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1"} err="failed to get container status \"89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1\": rpc error: code = NotFound desc = could not find container \"89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1\": container with ID starting with 89c8f6d9cdde2c66c3f98b4d69d51e9885f7bbbdc6697e0e418286370d1e63d1 not found: ID does not exist"
Jul 28 17:45:37.477226 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.477184    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf" podStartSLOduration=1.4771688570000001 podStartE2EDuration="1.477168857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:37.47633349 +0000 UTC m=+1378.130828465" watchObservedRunningTime="2026-07-28 17:45:37.477168857 +0000 UTC m=+1378.131663831"
Jul 28 17:45:37.491496 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.491471    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"]
Jul 28 17:45:37.493255 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.493236    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schel9txl"]
Jul 28 17:45:37.832836 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:37.832759    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13902b3c-5a03-4a81-b62a-232f6798a58b" path="/var/lib/kubelet/pods/13902b3c-5a03-4a81-b62a-232f6798a58b/volumes"
Jul 28 17:45:39.675354 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:39.675320    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:41.554377 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:41.554342    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:41.649378 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:41.649342    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:45.835508 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:45.835466    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:45.908915 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:45.908865    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:48.210646 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:48.210609    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:50.961559 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:50.961516    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:53.848297 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:53.848262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:54.087686 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:54.087647    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:54.996818 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:54.996777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:55.125134 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:45:55.125099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:08.460081 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:08.460049    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:46:09.073964 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:09.073932    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:20.879578 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:20.879540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:30.034924 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:30.034867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:30.105729 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:30.105697    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:36.410815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:36.410774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:36.494363 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:36.494333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:50.085655 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:50.085608    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:50.187442 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:50.187403    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:56.469196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:56.469162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:58.043467 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.043428    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:58.307405 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.307323    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"]
Jul 28 17:46:58.307630 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.307618    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13902b3c-5a03-4a81-b62a-232f6798a58b" containerName="main"
Jul 28 17:46:58.307677 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.307632    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="13902b3c-5a03-4a81-b62a-232f6798a58b" containerName="main"
Jul 28 17:46:58.307721 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.307689    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="13902b3c-5a03-4a81-b62a-232f6798a58b" containerName="main"
Jul 28 17:46:58.310531 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.310514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:46:58.316297 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.316267    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 28 17:46:58.317335 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.317012    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 28 17:46:58.317422 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.317356    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-lljr9\""
Jul 28 17:46:58.317524 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.317506    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-sfpqc\""
Jul 28 17:46:58.317683 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.317667    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 28 17:46:58.322142 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.322114    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"]
Jul 28 17:46:58.326905 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.326856    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:58.417437 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.417396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd495\" (UniqueName: \"kubernetes.io/projected/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-kube-api-access-pd495\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx\" (UID: \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:46:58.417618 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.417448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx\" (UID: \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:46:58.518620 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.518574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd495\" (UniqueName: \"kubernetes.io/projected/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-kube-api-access-pd495\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx\" (UID: \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:46:58.518812 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.518632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx\" (UID: \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:46:58.521381 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.521349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx\" (UID: \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:46:58.527744 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.527715    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd495\" (UniqueName: \"kubernetes.io/projected/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-kube-api-access-pd495\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx\" (UID: \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:46:58.622010 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.621975    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:46:58.755834 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.755807    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"]
Jul 28 17:46:58.757584 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:46:58.757542    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcde306f_95bb_4bf1_98b5_8c066e4f85ac.slice/crio-821bbbb4a3f27e3cc268f902d32b9533fabd5797e20b90859d2814471cb8eae0 WatchSource:0}: Error finding container 821bbbb4a3f27e3cc268f902d32b9533fabd5797e20b90859d2814471cb8eae0: Status 404 returned error can't find the container with id 821bbbb4a3f27e3cc268f902d32b9533fabd5797e20b90859d2814471cb8eae0
Jul 28 17:46:58.804207 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:58.804181    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx" event={"ID":"dcde306f-95bb-4bf1-98b5-8c066e4f85ac","Type":"ContainerStarted","Data":"821bbbb4a3f27e3cc268f902d32b9533fabd5797e20b90859d2814471cb8eae0"}
Jul 28 17:46:59.810430 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:59.810396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx" event={"ID":"dcde306f-95bb-4bf1-98b5-8c066e4f85ac","Type":"ContainerStarted","Data":"2740c0011d24c58849f4f266a866ba6fc8cbb34c1936b9f9078ec2a936b4def9"}
Jul 28 17:46:59.810833 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:59.810497    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:46:59.829830 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:46:59.829773    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx" podStartSLOduration=1.8297544270000001 podStartE2EDuration="1.829754427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:59.828637802 +0000 UTC m=+1460.483132779" watchObservedRunningTime="2026-07-28 17:46:59.829754427 +0000 UTC m=+1460.484249404"
Jul 28 17:47:02.829852 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:02.829820    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:02.935841 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:02.935794    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:07.675377 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:07.675339    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:30.817084 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:30.817042    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx" podUID="dcde306f-95bb-4bf1-98b5-8c066e4f85ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:47:31.849633 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:31.849584    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"]
Jul 28 17:47:31.851559 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:31.851522    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx" podUID="dcde306f-95bb-4bf1-98b5-8c066e4f85ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:47:31.852735 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:31.852707    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:31.973677 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:31.973621    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:41.852043 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:41.852004    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx" podUID="dcde306f-95bb-4bf1-98b5-8c066e4f85ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:47:46.874915 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:46.874854    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx" podUID="dcde306f-95bb-4bf1-98b5-8c066e4f85ac" containerName="main" containerID="cri-o://2740c0011d24c58849f4f266a866ba6fc8cbb34c1936b9f9078ec2a936b4def9" gracePeriod=45
Jul 28 17:47:47.012820 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:47.012782    2559 generic.go:356] "Generic (PLEG): container finished" podID="dcde306f-95bb-4bf1-98b5-8c066e4f85ac" containerID="2740c0011d24c58849f4f266a866ba6fc8cbb34c1936b9f9078ec2a936b4def9" exitCode=0
Jul 28 17:47:47.012987 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:47.012847    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx" event={"ID":"dcde306f-95bb-4bf1-98b5-8c066e4f85ac","Type":"ContainerDied","Data":"2740c0011d24c58849f4f266a866ba6fc8cbb34c1936b9f9078ec2a936b4def9"}
Jul 28 17:47:47.123723 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:47.123692    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:47:47.321743 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:47.321639    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-tls-certs\") pod \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\" (UID: \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\") "
Jul 28 17:47:47.321743 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:47.321703    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd495\" (UniqueName: \"kubernetes.io/projected/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-kube-api-access-pd495\") pod \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\" (UID: \"dcde306f-95bb-4bf1-98b5-8c066e4f85ac\") "
Jul 28 17:47:47.324078 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:47.324048    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcde306f-95bb-4bf1-98b5-8c066e4f85ac" (UID: "dcde306f-95bb-4bf1-98b5-8c066e4f85ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:47.324168 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:47.324132    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-kube-api-access-pd495" (OuterVolumeSpecName: "kube-api-access-pd495") pod "dcde306f-95bb-4bf1-98b5-8c066e4f85ac" (UID: "dcde306f-95bb-4bf1-98b5-8c066e4f85ac"). InnerVolumeSpecName "kube-api-access-pd495". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:47.422698 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:47.422644    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:47:47.422698 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:47.422678    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd495\" (UniqueName: \"kubernetes.io/projected/dcde306f-95bb-4bf1-98b5-8c066e4f85ac-kube-api-access-pd495\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:47:48.017990 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:48.017952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx" event={"ID":"dcde306f-95bb-4bf1-98b5-8c066e4f85ac","Type":"ContainerDied","Data":"821bbbb4a3f27e3cc268f902d32b9533fabd5797e20b90859d2814471cb8eae0"}
Jul 28 17:47:48.017990 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:48.017996    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"
Jul 28 17:47:48.018585 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:48.018026    2559 scope.go:117] "RemoveContainer" containerID="2740c0011d24c58849f4f266a866ba6fc8cbb34c1936b9f9078ec2a936b4def9"
Jul 28 17:47:48.034832 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:48.034806    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"]
Jul 28 17:47:48.041955 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:48.041930    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf6s6rx"]
Jul 28 17:47:49.832010 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:49.831978    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcde306f-95bb-4bf1-98b5-8c066e4f85ac" path="/var/lib/kubelet/pods/dcde306f-95bb-4bf1-98b5-8c066e4f85ac/volumes"
Jul 28 17:47:51.293202 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:51.293165    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:51.391932 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:51.391860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:51.522525 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:51.522482    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"]
Jul 28 17:47:51.523265 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:51.523229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:51.615441 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:51.615411    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:53.939831 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:53.939788    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:54.219152 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:54.219069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:58.460868 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:58.460829    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf" podUID="2337a18f-c119-4af1-8c17-082272711207" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:47:58.719133 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:47:58.719042    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:03.212429 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:03.212324    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:05.170660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:05.170616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:05.251072 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:05.251033    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:06.542296 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:06.542238    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf" podUID="2337a18f-c119-4af1-8c17-082272711207" containerName="main" containerID="cri-o://0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99" gracePeriod=45
Jul 28 17:48:06.792964 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:06.792895    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:48:06.867168 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:06.867134    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfcmw\" (UniqueName: \"kubernetes.io/projected/2337a18f-c119-4af1-8c17-082272711207-kube-api-access-xfcmw\") pod \"2337a18f-c119-4af1-8c17-082272711207\" (UID: \"2337a18f-c119-4af1-8c17-082272711207\") "
Jul 28 17:48:06.867337 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:06.867187    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2337a18f-c119-4af1-8c17-082272711207-tls-certs\") pod \"2337a18f-c119-4af1-8c17-082272711207\" (UID: \"2337a18f-c119-4af1-8c17-082272711207\") "
Jul 28 17:48:06.869487 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:06.869456    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2337a18f-c119-4af1-8c17-082272711207-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2337a18f-c119-4af1-8c17-082272711207" (UID: "2337a18f-c119-4af1-8c17-082272711207"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:06.869487 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:06.869474    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2337a18f-c119-4af1-8c17-082272711207-kube-api-access-xfcmw" (OuterVolumeSpecName: "kube-api-access-xfcmw") pod "2337a18f-c119-4af1-8c17-082272711207" (UID: "2337a18f-c119-4af1-8c17-082272711207"). InnerVolumeSpecName "kube-api-access-xfcmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:06.968677 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:06.968639    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2337a18f-c119-4af1-8c17-082272711207-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:48:06.968677 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:06.968672    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcmw\" (UniqueName: \"kubernetes.io/projected/2337a18f-c119-4af1-8c17-082272711207-kube-api-access-xfcmw\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:48:07.104312 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.104228    2559 generic.go:356] "Generic (PLEG): container finished" podID="2337a18f-c119-4af1-8c17-082272711207" containerID="0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99" exitCode=0
Jul 28 17:48:07.104312 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.104275    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"
Jul 28 17:48:07.104312 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.104302    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf" event={"ID":"2337a18f-c119-4af1-8c17-082272711207","Type":"ContainerDied","Data":"0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99"}
Jul 28 17:48:07.104553 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.104348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf" event={"ID":"2337a18f-c119-4af1-8c17-082272711207","Type":"ContainerDied","Data":"dcbd395bc094af55d4b5d0e3c35c05e710162558f719d338e1d7d16fed3e597b"}
Jul 28 17:48:07.104553 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.104369    2559 scope.go:117] "RemoveContainer" containerID="0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99"
Jul 28 17:48:07.112847 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.112829    2559 scope.go:117] "RemoveContainer" containerID="0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99"
Jul 28 17:48:07.113126 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:48:07.113106    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99\": container with ID starting with 0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99 not found: ID does not exist" containerID="0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99"
Jul 28 17:48:07.113183 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.113134    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99"} err="failed to get container status \"0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99\": rpc error: code = NotFound desc = could not find container \"0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99\": container with ID starting with 0f880b6be798aef65a92e384c71fd3a2beb52c1bf0369cd2c886b4151ffe1a99 not found: ID does not exist"
Jul 28 17:48:07.126549 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.126520    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"]
Jul 28 17:48:07.130422 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.130395    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-f8xbf"]
Jul 28 17:48:07.832543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:07.832514    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2337a18f-c119-4af1-8c17-082272711207" path="/var/lib/kubelet/pods/2337a18f-c119-4af1-8c17-082272711207/volumes"
Jul 28 17:48:11.621763 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:11.621733    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:11.718728 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:11.718691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:11.823544 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:11.823505    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:15.735097 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:15.735061    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:16.023532 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:16.023440    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:18.771997 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:18.771959    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:20.659000 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.658960    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:20.951601 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.951515    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"]
Jul 28 17:48:20.951853 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.951836    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2337a18f-c119-4af1-8c17-082272711207" containerName="main"
Jul 28 17:48:20.951853 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.951855    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2337a18f-c119-4af1-8c17-082272711207" containerName="main"
Jul 28 17:48:20.952030 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.951863    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcde306f-95bb-4bf1-98b5-8c066e4f85ac" containerName="main"
Jul 28 17:48:20.952030 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.951869    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcde306f-95bb-4bf1-98b5-8c066e4f85ac" containerName="main"
Jul 28 17:48:20.952030 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.951954    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2337a18f-c119-4af1-8c17-082272711207" containerName="main"
Jul 28 17:48:20.952030 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.951962    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcde306f-95bb-4bf1-98b5-8c066e4f85ac" containerName="main"
Jul 28 17:48:20.954801 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.954785    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:20.957706 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.957678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 28 17:48:20.957835 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.957819    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 28 17:48:20.958585 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.958566    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-8cfh9\""
Jul 28 17:48:20.958675 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.958566    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 28 17:48:20.958675 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.958616    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-epp-sa-dockercfg-2kjqp\""
Jul 28 17:48:20.974253 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:20.974224    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"]
Jul 28 17:48:21.017144 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:21.017111    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:21.086089 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:21.086055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb2f85-d702-421f-bf86-96324a03c935-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq\" (UID: \"1edb2f85-d702-421f-bf86-96324a03c935\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:21.086276 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:21.086121    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sndc\" (UniqueName: \"kubernetes.io/projected/1edb2f85-d702-421f-bf86-96324a03c935-kube-api-access-9sndc\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq\" (UID: \"1edb2f85-d702-421f-bf86-96324a03c935\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:21.186790 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:21.186752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb2f85-d702-421f-bf86-96324a03c935-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq\" (UID: \"1edb2f85-d702-421f-bf86-96324a03c935\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:21.186974 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:21.186833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sndc\" (UniqueName: \"kubernetes.io/projected/1edb2f85-d702-421f-bf86-96324a03c935-kube-api-access-9sndc\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq\" (UID: \"1edb2f85-d702-421f-bf86-96324a03c935\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:21.189305 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:21.189279    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb2f85-d702-421f-bf86-96324a03c935-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq\" (UID: \"1edb2f85-d702-421f-bf86-96324a03c935\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:21.197339 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:21.197311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sndc\" (UniqueName: \"kubernetes.io/projected/1edb2f85-d702-421f-bf86-96324a03c935-kube-api-access-9sndc\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq\" (UID: \"1edb2f85-d702-421f-bf86-96324a03c935\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:21.265559 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:21.265461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:21.411923 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:21.411723    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"]
Jul 28 17:48:21.414603 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:48:21.414573    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edb2f85_d702_421f_bf86_96324a03c935.slice/crio-0befea57b3f519c5d83e40d74b17fddc853e1b00311bb17166385eafe557bf3a WatchSource:0}: Error finding container 0befea57b3f519c5d83e40d74b17fddc853e1b00311bb17166385eafe557bf3a: Status 404 returned error can't find the container with id 0befea57b3f519c5d83e40d74b17fddc853e1b00311bb17166385eafe557bf3a
Jul 28 17:48:22.167164 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:22.167124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq" event={"ID":"1edb2f85-d702-421f-bf86-96324a03c935","Type":"ContainerStarted","Data":"1ad5188e441d2aca48bd27ca4b6cbd457401078f38981a828c6e876c90b51b8b"}
Jul 28 17:48:22.167554 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:22.167180    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq" event={"ID":"1edb2f85-d702-421f-bf86-96324a03c935","Type":"ContainerStarted","Data":"0befea57b3f519c5d83e40d74b17fddc853e1b00311bb17166385eafe557bf3a"}
Jul 28 17:48:22.167554 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:22.167218    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:22.238836 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:22.238783    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq" podStartSLOduration=2.238767279 podStartE2EDuration="2.238767279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:22.236594713 +0000 UTC m=+1542.891089669" watchObservedRunningTime="2026-07-28 17:48:22.238767279 +0000 UTC m=+1542.893262290"
Jul 28 17:48:25.275605 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:25.275559    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:25.382558 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:25.382526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:33.666152 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:33.666105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:37.326513 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:37.326474    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:37.455418 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:37.455378    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:53.174173 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:53.174142    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 17:48:56.808415 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:56.808378    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:58.096061 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:48:58.096023    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:01.858196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.858164    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"]
Jul 28 17:49:01.863246 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.863220    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:01.866703 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.866679    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 17:49:01.866893 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.866706    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 17:49:01.866893 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.866706    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 17:49:01.866893 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.866679    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-lvv7g\""
Jul 28 17:49:01.874029 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.874006    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"]
Jul 28 17:49:01.918868 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.918830    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:01.929185 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.929156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:01.929329 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.929190    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:01.929329 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.929244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:01.929329 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.929286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:01.929329 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.929313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:01.929479 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:01.929333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbppd\" (UniqueName: \"kubernetes.io/projected/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-kube-api-access-gbppd\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.030215 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.030185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.030421 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.030231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.030421 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.030263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbppd\" (UniqueName: \"kubernetes.io/projected/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-kube-api-access-gbppd\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.030421 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.030326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.030607 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.030447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.030607 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.030497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.030713 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.030647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.030803 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.030778    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.030846 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.030785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.032741 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.032718    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.032972 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.032955    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.039336 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.039311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbppd\" (UniqueName: \"kubernetes.io/projected/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-kube-api-access-gbppd\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.174368 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.174334    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:02.211916 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.211862    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:02.311270 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.311239    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"]
Jul 28 17:49:02.314540 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:49:02.314514    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa876ce_a920_44d8_b76f_4e4d03b96ba4.slice/crio-d69d4197afdfda2541932b3e88139302e4675835e8c8f888555f7ab672e604bb WatchSource:0}: Error finding container d69d4197afdfda2541932b3e88139302e4675835e8c8f888555f7ab672e604bb: Status 404 returned error can't find the container with id d69d4197afdfda2541932b3e88139302e4675835e8c8f888555f7ab672e604bb
Jul 28 17:49:02.316262 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.316248    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:49:02.336158 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:02.336125    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6" event={"ID":"4fa876ce-a920-44d8-b76f-4e4d03b96ba4","Type":"ContainerStarted","Data":"d69d4197afdfda2541932b3e88139302e4675835e8c8f888555f7ab672e604bb"}
Jul 28 17:49:04.348172 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:04.348137    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6" event={"ID":"4fa876ce-a920-44d8-b76f-4e4d03b96ba4","Type":"ContainerStarted","Data":"4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f"}
Jul 28 17:49:04.365524 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:04.365465    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6" podStartSLOduration=1.8326115330000001 podStartE2EDuration="3.365449628s" totalImagesPullingTime="1.532838095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:49:04.364099068 +0000 UTC m=+1585.018594042" watchObservedRunningTime="2026-07-28 17:49:04.365449628 +0000 UTC m=+1585.019944603"
Jul 28 17:49:04.584469 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:04.584439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:05.174962 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:05.174922    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:10.482435 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:10.482389    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:10.582701 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:10.582649    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:12.174531 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:12.174492    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:12.187089 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:12.187063    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:12.395479 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:12.395451    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:49:16.932021 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:16.931991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:54.969419 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:49:54.969378    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:36.105410 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.105370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:36.177703 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.177666    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"]
Jul 28 17:50:36.186696 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:50:36.186658    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:50:36.186696 ip-10-0-133-142 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:50:36.186696 ip-10-0-133-142 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 28 17:50:36.186696 ip-10-0-133-142 kubenswrapper[2559]:  > containerID="4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f" cmd=["/bin/sleep","15"]
Jul 28 17:50:36.186932 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.186716    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6" message="" err=<
Jul 28 17:50:36.186932 ip-10-0-133-142 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:50:36.186932 ip-10-0-133-142 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 28 17:50:36.186932 ip-10-0-133-142 kubenswrapper[2559]:  >
Jul 28 17:50:36.186932 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:50:36.186728    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:50:36.186932 ip-10-0-133-142 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:50:36.186932 ip-10-0-133-142 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 28 17:50:36.186932 ip-10-0-133-142 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6" podUID="4fa876ce-a920-44d8-b76f-4e4d03b96ba4" containerName="main" containerID="cri-o://4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f"
Jul 28 17:50:36.186932 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.186794    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6" podUID="4fa876ce-a920-44d8-b76f-4e4d03b96ba4" containerName="main" containerID="cri-o://4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f" gracePeriod=60
Jul 28 17:50:36.254630 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.254599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:36.444527 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.444499    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:50:36.614486 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.614456    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-model-cache\") pod \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") "
Jul 28 17:50:36.614671 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.614527    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tls-certs\") pod \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") "
Jul 28 17:50:36.614671 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.614573    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tmp-dir\") pod \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") "
Jul 28 17:50:36.614671 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.614596    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbppd\" (UniqueName: \"kubernetes.io/projected/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-kube-api-access-gbppd\") pod \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") "
Jul 28 17:50:36.614671 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.614632    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-dshm\") pod \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") "
Jul 28 17:50:36.614920 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.614780    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fa876ce-a920-44d8-b76f-4e4d03b96ba4" (UID: "4fa876ce-a920-44d8-b76f-4e4d03b96ba4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:36.614920 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.614802    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-home\") pod \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\" (UID: \"4fa876ce-a920-44d8-b76f-4e4d03b96ba4\") "
Jul 28 17:50:36.614920 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.614852    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fa876ce-a920-44d8-b76f-4e4d03b96ba4" (UID: "4fa876ce-a920-44d8-b76f-4e4d03b96ba4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:36.615102 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.615081    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-home" (OuterVolumeSpecName: "home") pod "4fa876ce-a920-44d8-b76f-4e4d03b96ba4" (UID: "4fa876ce-a920-44d8-b76f-4e4d03b96ba4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:36.615158 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.615107    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-model-cache\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.615158 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.615128    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tmp-dir\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.617183 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.617155    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fa876ce-a920-44d8-b76f-4e4d03b96ba4" (UID: "4fa876ce-a920-44d8-b76f-4e4d03b96ba4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:50:36.617447 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.617427    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-dshm" (OuterVolumeSpecName: "dshm") pod "4fa876ce-a920-44d8-b76f-4e4d03b96ba4" (UID: "4fa876ce-a920-44d8-b76f-4e4d03b96ba4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:36.617507 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.617448    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-kube-api-access-gbppd" (OuterVolumeSpecName: "kube-api-access-gbppd") pod "4fa876ce-a920-44d8-b76f-4e4d03b96ba4" (UID: "4fa876ce-a920-44d8-b76f-4e4d03b96ba4"). InnerVolumeSpecName "kube-api-access-gbppd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:50:36.715831 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.715790    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbppd\" (UniqueName: \"kubernetes.io/projected/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-kube-api-access-gbppd\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.715831 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.715826    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-dshm\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.715831 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.715839    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-home\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.716179 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.715851    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa876ce-a920-44d8-b76f-4e4d03b96ba4-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.737136 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.737102    2559 generic.go:356] "Generic (PLEG): container finished" podID="4fa876ce-a920-44d8-b76f-4e4d03b96ba4" containerID="4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f" exitCode=0
Jul 28 17:50:36.737297 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.737144    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"
Jul 28 17:50:36.737297 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.737196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6" event={"ID":"4fa876ce-a920-44d8-b76f-4e4d03b96ba4","Type":"ContainerDied","Data":"4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f"}
Jul 28 17:50:36.737297 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.737247    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6" event={"ID":"4fa876ce-a920-44d8-b76f-4e4d03b96ba4","Type":"ContainerDied","Data":"d69d4197afdfda2541932b3e88139302e4675835e8c8f888555f7ab672e604bb"}
Jul 28 17:50:36.737297 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.737268    2559 scope.go:117] "RemoveContainer" containerID="4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f"
Jul 28 17:50:36.748415 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.748390    2559 scope.go:117] "RemoveContainer" containerID="4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f"
Jul 28 17:50:36.748689 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:50:36.748671    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f\": container with ID starting with 4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f not found: ID does not exist" containerID="4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f"
Jul 28 17:50:36.748742 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.748698    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f"} err="failed to get container status \"4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f\": rpc error: code = NotFound desc = could not find container \"4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f\": container with ID starting with 4db0099ac30e3a1ee27d7a4187e32c4694de6e4cf900684496e02499cf3bc91f not found: ID does not exist"
Jul 28 17:50:36.772524 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.772490    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"]
Jul 28 17:50:36.785190 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:36.785157    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6"]
Jul 28 17:50:37.384971 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:37.384930    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vzqf6" podUID="4fa876ce-a920-44d8-b76f-4e4d03b96ba4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:50:37.833349 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:37.833254    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fa876ce-a920-44d8-b76f-4e4d03b96ba4" path="/var/lib/kubelet/pods/4fa876ce-a920-44d8-b76f-4e4d03b96ba4/volumes"
Jul 28 17:50:45.599542 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:45.599486    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:55.837361 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:55.837321    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:55.937446 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:50:55.937415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:02.917163 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:02.917043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:09.344407 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:09.344367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:09.439159 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:09.439120    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:12.368670 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.368632    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"]
Jul 28 17:51:12.369081 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.368971    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa876ce-a920-44d8-b76f-4e4d03b96ba4" containerName="main"
Jul 28 17:51:12.369081 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.368984    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa876ce-a920-44d8-b76f-4e4d03b96ba4" containerName="main"
Jul 28 17:51:12.369081 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.369045    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa876ce-a920-44d8-b76f-4e4d03b96ba4" containerName="main"
Jul 28 17:51:12.372191 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.372170    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.375259 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.375237    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 17:51:12.375817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.375792    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 17:51:12.375817 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.375808    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-5xhl9\""
Jul 28 17:51:12.376001 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.375792    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-58l4l\""
Jul 28 17:51:12.376001 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.375814    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 17:51:12.384812 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.384792    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"]
Jul 28 17:51:12.404657 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.404631    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:12.408845 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.408823    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198bcec6-15e6-470e-a74f-2aa940ff94a0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.408962 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.408901    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.408962 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.408931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.409087 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.408967    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vvk\" (UniqueName: \"kubernetes.io/projected/198bcec6-15e6-470e-a74f-2aa940ff94a0-kube-api-access-46vvk\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.409087 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.409035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.409087 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.409063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.510053 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.510022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.510053 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.510052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.510273 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.510076    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vvk\" (UniqueName: \"kubernetes.io/projected/198bcec6-15e6-470e-a74f-2aa940ff94a0-kube-api-access-46vvk\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.510273 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.510129    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.510273 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.510155    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.510273 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.510189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198bcec6-15e6-470e-a74f-2aa940ff94a0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.510486 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.510424    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.510486 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.510468    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.510605 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.510536    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.512535 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.512514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.512734 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.512716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198bcec6-15e6-470e-a74f-2aa940ff94a0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.518143 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.518120    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vvk\" (UniqueName: \"kubernetes.io/projected/198bcec6-15e6-470e-a74f-2aa940ff94a0-kube-api-access-46vvk\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.625006 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.624895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:12.681629 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.681595    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:12.825695 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.825626    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"]
Jul 28 17:51:12.828352 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:51:12.828327    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198bcec6_15e6_470e_a74f_2aa940ff94a0.slice/crio-c678ff7889a55f2e633bbcba2b31c0a8ab479618f450b2cfffc3b17257c79aed WatchSource:0}: Error finding container c678ff7889a55f2e633bbcba2b31c0a8ab479618f450b2cfffc3b17257c79aed: Status 404 returned error can't find the container with id c678ff7889a55f2e633bbcba2b31c0a8ab479618f450b2cfffc3b17257c79aed
Jul 28 17:51:12.889185 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:12.889102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" event={"ID":"198bcec6-15e6-470e-a74f-2aa940ff94a0","Type":"ContainerStarted","Data":"c678ff7889a55f2e633bbcba2b31c0a8ab479618f450b2cfffc3b17257c79aed"}
Jul 28 17:51:14.901288 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:14.901246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" event={"ID":"198bcec6-15e6-470e-a74f-2aa940ff94a0","Type":"ContainerStarted","Data":"01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca"}
Jul 28 17:51:14.901703 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:14.901391    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:15.907776 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:15.907735    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" event={"ID":"198bcec6-15e6-470e-a74f-2aa940ff94a0","Type":"ContainerStarted","Data":"ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c"}
Jul 28 17:51:15.941074 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:15.941020    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" podStartSLOduration=2.388375963 podStartE2EDuration="3.941004434s" totalImagesPullingTime="1.552628471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:51:15.939866297 +0000 UTC m=+1716.594361272" watchObservedRunningTime="2026-07-28 17:51:15.941004434 +0000 UTC m=+1716.595499409"
Jul 28 17:51:16.682206 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:16.682166    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:22.682749 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:22.682640    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:22.694894 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:22.694854    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:22.948962 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:22.948860    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:23.461197 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:23.461163    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:25.622843 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:25.622809    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:33.953947 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:33.953911    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:46.202460 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.202409    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"]
Jul 28 17:51:46.207292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.207246    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:46.212029 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:51:46.211997    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:51:46.212029 ip-10-0-133-142 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:51:46.212029 ip-10-0-133-142 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 28 17:51:46.212029 ip-10-0-133-142 kubenswrapper[2559]:  > containerID="ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c" cmd=["/bin/sleep","15"]
Jul 28 17:51:46.212239 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.212045    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" message="" err=<
Jul 28 17:51:46.212239 ip-10-0-133-142 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:51:46.212239 ip-10-0-133-142 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 28 17:51:46.212239 ip-10-0-133-142 kubenswrapper[2559]:  >
Jul 28 17:51:46.212239 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:51:46.212058    2559 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:51:46.212239 ip-10-0-133-142 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:51:46.212239 ip-10-0-133-142 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 28 17:51:46.212239 ip-10-0-133-142 kubenswrapper[2559]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerName="main" containerID="cri-o://ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c"
Jul 28 17:51:46.212239 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.212107    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerName="main" containerID="cri-o://ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c" gracePeriod=60
Jul 28 17:51:46.322000 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.321956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:46.428542 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.428505    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerName="llm-d-routing-sidecar" containerID="cri-o://01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca" gracePeriod=60
Jul 28 17:51:46.679196 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.679170    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:46.697728 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.697698    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-home\") pod \"198bcec6-15e6-470e-a74f-2aa940ff94a0\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") "
Jul 28 17:51:46.697893 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.697734    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-dshm\") pod \"198bcec6-15e6-470e-a74f-2aa940ff94a0\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") "
Jul 28 17:51:46.697893 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.697756    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46vvk\" (UniqueName: \"kubernetes.io/projected/198bcec6-15e6-470e-a74f-2aa940ff94a0-kube-api-access-46vvk\") pod \"198bcec6-15e6-470e-a74f-2aa940ff94a0\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") "
Jul 28 17:51:46.697893 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.697783    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-tmp-dir\") pod \"198bcec6-15e6-470e-a74f-2aa940ff94a0\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") "
Jul 28 17:51:46.697893 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.697800    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-model-cache\") pod \"198bcec6-15e6-470e-a74f-2aa940ff94a0\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") "
Jul 28 17:51:46.697893 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.697840    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198bcec6-15e6-470e-a74f-2aa940ff94a0-tls-certs\") pod \"198bcec6-15e6-470e-a74f-2aa940ff94a0\" (UID: \"198bcec6-15e6-470e-a74f-2aa940ff94a0\") "
Jul 28 17:51:46.698166 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.698013    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-home" (OuterVolumeSpecName: "home") pod "198bcec6-15e6-470e-a74f-2aa940ff94a0" (UID: "198bcec6-15e6-470e-a74f-2aa940ff94a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:46.698166 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.698106    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "198bcec6-15e6-470e-a74f-2aa940ff94a0" (UID: "198bcec6-15e6-470e-a74f-2aa940ff94a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:46.698166 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.698124    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "198bcec6-15e6-470e-a74f-2aa940ff94a0" (UID: "198bcec6-15e6-470e-a74f-2aa940ff94a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:46.700094 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.700074    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-dshm" (OuterVolumeSpecName: "dshm") pod "198bcec6-15e6-470e-a74f-2aa940ff94a0" (UID: "198bcec6-15e6-470e-a74f-2aa940ff94a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:46.700186 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.700090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198bcec6-15e6-470e-a74f-2aa940ff94a0-kube-api-access-46vvk" (OuterVolumeSpecName: "kube-api-access-46vvk") pod "198bcec6-15e6-470e-a74f-2aa940ff94a0" (UID: "198bcec6-15e6-470e-a74f-2aa940ff94a0"). InnerVolumeSpecName "kube-api-access-46vvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:51:46.700186 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.700143    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198bcec6-15e6-470e-a74f-2aa940ff94a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "198bcec6-15e6-470e-a74f-2aa940ff94a0" (UID: "198bcec6-15e6-470e-a74f-2aa940ff94a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:51:46.798897 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.798783    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-tmp-dir\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.798897 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.798814    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-model-cache\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.798897 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.798822    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198bcec6-15e6-470e-a74f-2aa940ff94a0-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.798897 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.798830    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-home\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.798897 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.798837    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198bcec6-15e6-470e-a74f-2aa940ff94a0-dshm\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.798897 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:46.798845    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46vvk\" (UniqueName: \"kubernetes.io/projected/198bcec6-15e6-470e-a74f-2aa940ff94a0-kube-api-access-46vvk\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:51:47.044108 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.044078    2559 generic.go:356] "Generic (PLEG): container finished" podID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerID="ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c" exitCode=0
Jul 28 17:51:47.044108 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.044101    2559 generic.go:356] "Generic (PLEG): container finished" podID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerID="01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca" exitCode=0
Jul 28 17:51:47.044307 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.044133    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" event={"ID":"198bcec6-15e6-470e-a74f-2aa940ff94a0","Type":"ContainerDied","Data":"ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c"}
Jul 28 17:51:47.044307 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.044172    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" event={"ID":"198bcec6-15e6-470e-a74f-2aa940ff94a0","Type":"ContainerDied","Data":"01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca"}
Jul 28 17:51:47.044307 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.044188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq" event={"ID":"198bcec6-15e6-470e-a74f-2aa940ff94a0","Type":"ContainerDied","Data":"c678ff7889a55f2e633bbcba2b31c0a8ab479618f450b2cfffc3b17257c79aed"}
Jul 28 17:51:47.044307 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.044133    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"
Jul 28 17:51:47.044307 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.044209    2559 scope.go:117] "RemoveContainer" containerID="ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c"
Jul 28 17:51:47.053184 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.053160    2559 scope.go:117] "RemoveContainer" containerID="01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca"
Jul 28 17:51:47.060560 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.060541    2559 scope.go:117] "RemoveContainer" containerID="ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c"
Jul 28 17:51:47.060805 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:51:47.060784    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c\": container with ID starting with ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c not found: ID does not exist" containerID="ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c"
Jul 28 17:51:47.060859 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.060823    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c"} err="failed to get container status \"ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c\": rpc error: code = NotFound desc = could not find container \"ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c\": container with ID starting with ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c not found: ID does not exist"
Jul 28 17:51:47.060859 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.060842    2559 scope.go:117] "RemoveContainer" containerID="01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca"
Jul 28 17:51:47.061086 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:51:47.061073    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca\": container with ID starting with 01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca not found: ID does not exist" containerID="01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca"
Jul 28 17:51:47.061133 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.061088    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca"} err="failed to get container status \"01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca\": rpc error: code = NotFound desc = could not find container \"01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca\": container with ID starting with 01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca not found: ID does not exist"
Jul 28 17:51:47.061133 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.061100    2559 scope.go:117] "RemoveContainer" containerID="ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c"
Jul 28 17:51:47.061292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.061276    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c"} err="failed to get container status \"ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c\": rpc error: code = NotFound desc = could not find container \"ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c\": container with ID starting with ab20386a17e20f4720dac7d05765254ccde03c258e6a139ad649ccfffe215c7c not found: ID does not exist"
Jul 28 17:51:47.061292 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.061292    2559 scope.go:117] "RemoveContainer" containerID="01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca"
Jul 28 17:51:47.061497 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.061477    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca"} err="failed to get container status \"01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca\": rpc error: code = NotFound desc = could not find container \"01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca\": container with ID starting with 01d4f125a8fdf579b15111572f3bf07e3dd738468f2788761715eca227753aca not found: ID does not exist"
Jul 28 17:51:47.088032 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.088007    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"]
Jul 28 17:51:47.092899 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.092861    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-dlxrq"]
Jul 28 17:51:47.832367 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:51:47.832331    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" path="/var/lib/kubelet/pods/198bcec6-15e6-470e-a74f-2aa940ff94a0/volumes"
Jul 28 17:52:06.199615 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:06.199577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:06.299109 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:06.299073    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:13.013239 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:13.013193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:17.805284 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:17.805245    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:18.196146 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.196107    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:18.222803 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.222772    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"]
Jul 28 17:52:18.223086 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.223074    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerName="llm-d-routing-sidecar"
Jul 28 17:52:18.223130 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.223087    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerName="llm-d-routing-sidecar"
Jul 28 17:52:18.223130 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.223099    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerName="main"
Jul 28 17:52:18.223130 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.223105    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerName="main"
Jul 28 17:52:18.223227 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.223153    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerName="main"
Jul 28 17:52:18.223227 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.223163    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="198bcec6-15e6-470e-a74f-2aa940ff94a0" containerName="llm-d-routing-sidecar"
Jul 28 17:52:18.225973 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.225957    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:52:18.235148 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.235123    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-zbtrr\""
Jul 28 17:52:18.235285 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.235151    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-fx57c\""
Jul 28 17:52:18.235285 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.235180    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 17:52:18.235285 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.235127    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 17:52:18.235285 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.235151    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 17:52:18.257121 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.257088    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"]
Jul 28 17:52:18.366390 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.366356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprpr\" (UniqueName: \"kubernetes.io/projected/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-kube-api-access-xprpr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp\" (UID: \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:52:18.366566 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.366422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp\" (UID: \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:52:18.404048 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.404008    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:18.467191 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.467100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprpr\" (UniqueName: \"kubernetes.io/projected/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-kube-api-access-xprpr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp\" (UID: \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:52:18.467191 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.467181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp\" (UID: \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:52:18.469750 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.469722    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp\" (UID: \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:52:18.481224 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.481195    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprpr\" (UniqueName: \"kubernetes.io/projected/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-kube-api-access-xprpr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp\" (UID: \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:52:18.536065 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.536032    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:52:18.697631 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:18.697517    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"]
Jul 28 17:52:18.700239 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:52:18.700211    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9594ff56_0af3_4d72_a62a_4fbf50e51d7e.slice/crio-80d26029181f20eb37d8ed0c8c5d10dde1d0da028fdcafb221ca2ebf8ec8273e WatchSource:0}: Error finding container 80d26029181f20eb37d8ed0c8c5d10dde1d0da028fdcafb221ca2ebf8ec8273e: Status 404 returned error can't find the container with id 80d26029181f20eb37d8ed0c8c5d10dde1d0da028fdcafb221ca2ebf8ec8273e
Jul 28 17:52:19.187059 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:19.187024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp" event={"ID":"9594ff56-0af3-4d72-a62a-4fbf50e51d7e","Type":"ContainerStarted","Data":"abbc66910dcc90b3a4a4bfcfead646ba44c1b62e1725f0d43ea0b4aa975d6bac"}
Jul 28 17:52:19.187511 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:19.187098    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp" event={"ID":"9594ff56-0af3-4d72-a62a-4fbf50e51d7e","Type":"ContainerStarted","Data":"80d26029181f20eb37d8ed0c8c5d10dde1d0da028fdcafb221ca2ebf8ec8273e"}
Jul 28 17:52:19.187511 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:19.187118    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:52:19.212341 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:19.212289    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp" podStartSLOduration=1.2122722320000001 podStartE2EDuration="1.212272232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:19.211594665 +0000 UTC m=+1779.866089642" watchObservedRunningTime="2026-07-28 17:52:19.212272232 +0000 UTC m=+1779.866767208"
Jul 28 17:52:20.663129 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:20.663092    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:20.758322 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:20.758279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:24.993917 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:24.993861    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:50.192325 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:52:50.192298    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:53:32.235438 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:32.235403    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:32.299975 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:32.299941    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"]
Jul 28 17:53:32.340660 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:32.340626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:32.421360 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:32.421321    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:40.192763 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:40.192727    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp" podUID="9594ff56-0af3-4d72-a62a-4fbf50e51d7e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:53:47.319624 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.319577    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp" podUID="9594ff56-0af3-4d72-a62a-4fbf50e51d7e" containerName="main" containerID="cri-o://abbc66910dcc90b3a4a4bfcfead646ba44c1b62e1725f0d43ea0b4aa975d6bac" gracePeriod=45
Jul 28 17:53:47.565389 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.565353    2559 generic.go:356] "Generic (PLEG): container finished" podID="9594ff56-0af3-4d72-a62a-4fbf50e51d7e" containerID="abbc66910dcc90b3a4a4bfcfead646ba44c1b62e1725f0d43ea0b4aa975d6bac" exitCode=0
Jul 28 17:53:47.565501 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.565416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp" event={"ID":"9594ff56-0af3-4d72-a62a-4fbf50e51d7e","Type":"ContainerDied","Data":"abbc66910dcc90b3a4a4bfcfead646ba44c1b62e1725f0d43ea0b4aa975d6bac"}
Jul 28 17:53:47.565501 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.565463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp" event={"ID":"9594ff56-0af3-4d72-a62a-4fbf50e51d7e","Type":"ContainerDied","Data":"80d26029181f20eb37d8ed0c8c5d10dde1d0da028fdcafb221ca2ebf8ec8273e"}
Jul 28 17:53:47.565501 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.565487    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d26029181f20eb37d8ed0c8c5d10dde1d0da028fdcafb221ca2ebf8ec8273e"
Jul 28 17:53:47.572751 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.572700    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:53:47.598191 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.598159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-tls-certs\") pod \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\" (UID: \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\") "
Jul 28 17:53:47.598335 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.598228    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xprpr\" (UniqueName: \"kubernetes.io/projected/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-kube-api-access-xprpr\") pod \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\" (UID: \"9594ff56-0af3-4d72-a62a-4fbf50e51d7e\") "
Jul 28 17:53:47.600517 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.600490    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-kube-api-access-xprpr" (OuterVolumeSpecName: "kube-api-access-xprpr") pod "9594ff56-0af3-4d72-a62a-4fbf50e51d7e" (UID: "9594ff56-0af3-4d72-a62a-4fbf50e51d7e"). InnerVolumeSpecName "kube-api-access-xprpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:47.600630 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.600518    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9594ff56-0af3-4d72-a62a-4fbf50e51d7e" (UID: "9594ff56-0af3-4d72-a62a-4fbf50e51d7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:47.699480 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.699446    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:53:47.699480 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:47.699474    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xprpr\" (UniqueName: \"kubernetes.io/projected/9594ff56-0af3-4d72-a62a-4fbf50e51d7e-kube-api-access-xprpr\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:53:48.569628 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:48.569596    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"
Jul 28 17:53:48.588572 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:48.588544    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"]
Jul 28 17:53:48.593169 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:48.593146    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqfvp"]
Jul 28 17:53:49.834415 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:49.834382    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9594ff56-0af3-4d72-a62a-4fbf50e51d7e" path="/var/lib/kubelet/pods/9594ff56-0af3-4d72-a62a-4fbf50e51d7e/volumes"
Jul 28 17:53:52.203483 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:52.203443    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:52.282278 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:52.282245    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:57.299638 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.299602    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:57.593250 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.593140    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"]
Jul 28 17:53:57.593612 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.593595    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9594ff56-0af3-4d72-a62a-4fbf50e51d7e" containerName="main"
Jul 28 17:53:57.593725 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.593615    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9594ff56-0af3-4d72-a62a-4fbf50e51d7e" containerName="main"
Jul 28 17:53:57.593725 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.593694    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9594ff56-0af3-4d72-a62a-4fbf50e51d7e" containerName="main"
Jul 28 17:53:57.597084 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.597063    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:53:57.599929 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.599900    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:57.600626 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.600594    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-lpvmq\""
Jul 28 17:53:57.600733 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.600596    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 17:53:57.600733 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.600637    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 17:53:57.600733 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.600637    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-9vvfm\""
Jul 28 17:53:57.600957 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.600596    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 17:53:57.621492 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.621470    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"]
Jul 28 17:53:57.691316 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.691287    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e263ba34-1f42-4984-8830-e13be14a222d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd\" (UID: \"e263ba34-1f42-4984-8830-e13be14a222d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:53:57.691442 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.691343    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrgz\" (UniqueName: \"kubernetes.io/projected/e263ba34-1f42-4984-8830-e13be14a222d-kube-api-access-jkrgz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd\" (UID: \"e263ba34-1f42-4984-8830-e13be14a222d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:53:57.697109 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.697081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:57.792060 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.792023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrgz\" (UniqueName: \"kubernetes.io/projected/e263ba34-1f42-4984-8830-e13be14a222d-kube-api-access-jkrgz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd\" (UID: \"e263ba34-1f42-4984-8830-e13be14a222d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:53:57.792263 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.792080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e263ba34-1f42-4984-8830-e13be14a222d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd\" (UID: \"e263ba34-1f42-4984-8830-e13be14a222d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:53:57.794641 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.794615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e263ba34-1f42-4984-8830-e13be14a222d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd\" (UID: \"e263ba34-1f42-4984-8830-e13be14a222d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:53:57.803486 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.803459    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrgz\" (UniqueName: \"kubernetes.io/projected/e263ba34-1f42-4984-8830-e13be14a222d-kube-api-access-jkrgz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd\" (UID: \"e263ba34-1f42-4984-8830-e13be14a222d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:53:57.907318 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:57.907284    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:53:58.056973 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:58.056945    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"]
Jul 28 17:53:58.057488 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:53:58.057468    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode263ba34_1f42_4984_8830_e13be14a222d.slice/crio-1762b74393182e0d1d9461019434e7252aab706abf3d80d722d6d9d48f3bb4bc WatchSource:0}: Error finding container 1762b74393182e0d1d9461019434e7252aab706abf3d80d722d6d9d48f3bb4bc: Status 404 returned error can't find the container with id 1762b74393182e0d1d9461019434e7252aab706abf3d80d722d6d9d48f3bb4bc
Jul 28 17:53:58.612862 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:58.612828    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd" event={"ID":"e263ba34-1f42-4984-8830-e13be14a222d","Type":"ContainerStarted","Data":"e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69"}
Jul 28 17:53:58.613353 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:58.612891    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd" event={"ID":"e263ba34-1f42-4984-8830-e13be14a222d","Type":"ContainerStarted","Data":"1762b74393182e0d1d9461019434e7252aab706abf3d80d722d6d9d48f3bb4bc"}
Jul 28 17:53:58.613353 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:58.612915    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:53:58.636002 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:53:58.635949    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd" podStartSLOduration=1.6359300110000001 podStartE2EDuration="1.635930011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:58.634130034 +0000 UTC m=+1879.288625009" watchObservedRunningTime="2026-07-28 17:53:58.635930011 +0000 UTC m=+1879.290425025"
Jul 28 17:54:00.301045 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:00.301011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:06.254289 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:06.254251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:06.358262 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:06.358214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:10.780748 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:10.780704    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:19.394419 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:19.394367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:19.520286 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:19.520250    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:29.618793 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:29.618763    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:54:37.230433 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:37.230399    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"]
Jul 28 17:54:37.277743 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:37.277707    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:37.351269 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:37.351234    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:39.619063 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:39.619022    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd" podUID="e263ba34-1f42-4984-8830-e13be14a222d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:54:49.619183 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:49.619142    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd" podUID="e263ba34-1f42-4984-8830-e13be14a222d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:54:52.251099 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.251055    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd" podUID="e263ba34-1f42-4984-8830-e13be14a222d" containerName="main" containerID="cri-o://e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69" gracePeriod=45
Jul 28 17:54:52.496289 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.496266    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:54:52.546781 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.546715    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e263ba34-1f42-4984-8830-e13be14a222d-tls-certs\") pod \"e263ba34-1f42-4984-8830-e13be14a222d\" (UID: \"e263ba34-1f42-4984-8830-e13be14a222d\") "
Jul 28 17:54:52.546781 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.546749    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkrgz\" (UniqueName: \"kubernetes.io/projected/e263ba34-1f42-4984-8830-e13be14a222d-kube-api-access-jkrgz\") pod \"e263ba34-1f42-4984-8830-e13be14a222d\" (UID: \"e263ba34-1f42-4984-8830-e13be14a222d\") "
Jul 28 17:54:52.548983 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.548955    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e263ba34-1f42-4984-8830-e13be14a222d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e263ba34-1f42-4984-8830-e13be14a222d" (UID: "e263ba34-1f42-4984-8830-e13be14a222d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:54:52.549101 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.549039    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e263ba34-1f42-4984-8830-e13be14a222d-kube-api-access-jkrgz" (OuterVolumeSpecName: "kube-api-access-jkrgz") pod "e263ba34-1f42-4984-8830-e13be14a222d" (UID: "e263ba34-1f42-4984-8830-e13be14a222d"). InnerVolumeSpecName "kube-api-access-jkrgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:54:52.647606 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.647575    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e263ba34-1f42-4984-8830-e13be14a222d-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:54:52.647606 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.647603    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrgz\" (UniqueName: \"kubernetes.io/projected/e263ba34-1f42-4984-8830-e13be14a222d-kube-api-access-jkrgz\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:54:52.836172 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.836064    2559 generic.go:356] "Generic (PLEG): container finished" podID="e263ba34-1f42-4984-8830-e13be14a222d" containerID="e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69" exitCode=0
Jul 28 17:54:52.836172 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.836101    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"
Jul 28 17:54:52.836172 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.836108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd" event={"ID":"e263ba34-1f42-4984-8830-e13be14a222d","Type":"ContainerDied","Data":"e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69"}
Jul 28 17:54:52.836172 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.836158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd" event={"ID":"e263ba34-1f42-4984-8830-e13be14a222d","Type":"ContainerDied","Data":"1762b74393182e0d1d9461019434e7252aab706abf3d80d722d6d9d48f3bb4bc"}
Jul 28 17:54:52.836498 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.836188    2559 scope.go:117] "RemoveContainer" containerID="e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69"
Jul 28 17:54:52.845102 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.845084    2559 scope.go:117] "RemoveContainer" containerID="e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69"
Jul 28 17:54:52.845369 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:54:52.845344    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69\": container with ID starting with e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69 not found: ID does not exist" containerID="e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69"
Jul 28 17:54:52.845421 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.845372    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69"} err="failed to get container status \"e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69\": rpc error: code = NotFound desc = could not find container \"e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69\": container with ID starting with e475434bb7795f3b00898fec5832381fe9c019a4355c337cc6f8775970b27d69 not found: ID does not exist"
Jul 28 17:54:52.858506 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.858481    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"]
Jul 28 17:54:52.862359 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:52.862336    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b9c25sd"]
Jul 28 17:54:53.832477 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:53.832435    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e263ba34-1f42-4984-8830-e13be14a222d" path="/var/lib/kubelet/pods/e263ba34-1f42-4984-8830-e13be14a222d/volumes"
Jul 28 17:54:57.317201 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:57.317170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:57.424307 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:57.424270    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:57.525104 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:54:57.525067    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:02.924339 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:02.924302    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:03.081469 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.081435    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"]
Jul 28 17:55:03.081747 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.081736    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e263ba34-1f42-4984-8830-e13be14a222d" containerName="main"
Jul 28 17:55:03.081794 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.081749    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e263ba34-1f42-4984-8830-e13be14a222d" containerName="main"
Jul 28 17:55:03.081842 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.081802    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e263ba34-1f42-4984-8830-e13be14a222d" containerName="main"
Jul 28 17:55:03.085940 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.085918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:03.089161 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.089127    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-cmcck\""
Jul 28 17:55:03.089299 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.089174    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 28 17:55:03.089299 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.089174    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-6p8gb\""
Jul 28 17:55:03.089299 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.089277    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 28 17:55:03.089488 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.089424    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 28 17:55:03.094815 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.094792    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"]
Jul 28 17:55:03.144983 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.144952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:03.237051 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.236959    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b27ab1-a39e-4400-99cf-6fde93ef4782-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk\" (UID: \"44b27ab1-a39e-4400-99cf-6fde93ef4782\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:03.237051 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.237014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl89r\" (UniqueName: \"kubernetes.io/projected/44b27ab1-a39e-4400-99cf-6fde93ef4782-kube-api-access-pl89r\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk\" (UID: \"44b27ab1-a39e-4400-99cf-6fde93ef4782\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:03.338423 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.338392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b27ab1-a39e-4400-99cf-6fde93ef4782-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk\" (UID: \"44b27ab1-a39e-4400-99cf-6fde93ef4782\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:03.338592 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.338441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl89r\" (UniqueName: \"kubernetes.io/projected/44b27ab1-a39e-4400-99cf-6fde93ef4782-kube-api-access-pl89r\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk\" (UID: \"44b27ab1-a39e-4400-99cf-6fde93ef4782\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:03.341125 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.341099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b27ab1-a39e-4400-99cf-6fde93ef4782-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk\" (UID: \"44b27ab1-a39e-4400-99cf-6fde93ef4782\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:03.348020 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.347991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl89r\" (UniqueName: \"kubernetes.io/projected/44b27ab1-a39e-4400-99cf-6fde93ef4782-kube-api-access-pl89r\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk\" (UID: \"44b27ab1-a39e-4400-99cf-6fde93ef4782\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:03.396277 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.396242    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:03.532368 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.532294    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"]
Jul 28 17:55:03.535026 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:55:03.535004    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b27ab1_a39e_4400_99cf_6fde93ef4782.slice/crio-b764fc32185bea7b35f197c1033c87c380b319d417c37b6d2eaf92020079478b WatchSource:0}: Error finding container b764fc32185bea7b35f197c1033c87c380b319d417c37b6d2eaf92020079478b: Status 404 returned error can't find the container with id b764fc32185bea7b35f197c1033c87c380b319d417c37b6d2eaf92020079478b
Jul 28 17:55:03.881891 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.881832    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk" event={"ID":"44b27ab1-a39e-4400-99cf-6fde93ef4782","Type":"ContainerStarted","Data":"19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da"}
Jul 28 17:55:03.882100 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.881914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk" event={"ID":"44b27ab1-a39e-4400-99cf-6fde93ef4782","Type":"ContainerStarted","Data":"b764fc32185bea7b35f197c1033c87c380b319d417c37b6d2eaf92020079478b"}
Jul 28 17:55:03.882100 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.881947    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:03.910946 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:03.910888    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk" podStartSLOduration=0.910854991 podStartE2EDuration="910.854991ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:55:03.910384339 +0000 UTC m=+1944.564879315" watchObservedRunningTime="2026-07-28 17:55:03.910854991 +0000 UTC m=+1944.565349969"
Jul 28 17:55:05.552389 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:05.552351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:11.673966 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:11.673930    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:11.775903 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:11.775842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:18.903972 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:18.903931    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:34.887962 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:34.887930    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:36.238578 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:36.238543    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"]
Jul 28 17:55:36.242543 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:36.242507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:36.321390 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:36.321355    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:44.888010 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:44.887970    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk" podUID="44b27ab1-a39e-4400-99cf-6fde93ef4782" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:55:51.258677 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:51.258623    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk" podUID="44b27ab1-a39e-4400-99cf-6fde93ef4782" containerName="main" containerID="cri-o://19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da" gracePeriod=45
Jul 28 17:55:51.504373 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:51.504346    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:51.650504 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:51.650475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl89r\" (UniqueName: \"kubernetes.io/projected/44b27ab1-a39e-4400-99cf-6fde93ef4782-kube-api-access-pl89r\") pod \"44b27ab1-a39e-4400-99cf-6fde93ef4782\" (UID: \"44b27ab1-a39e-4400-99cf-6fde93ef4782\") "
Jul 28 17:55:51.650696 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:51.650542    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b27ab1-a39e-4400-99cf-6fde93ef4782-tls-certs\") pod \"44b27ab1-a39e-4400-99cf-6fde93ef4782\" (UID: \"44b27ab1-a39e-4400-99cf-6fde93ef4782\") "
Jul 28 17:55:51.652846 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:51.652813    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b27ab1-a39e-4400-99cf-6fde93ef4782-kube-api-access-pl89r" (OuterVolumeSpecName: "kube-api-access-pl89r") pod "44b27ab1-a39e-4400-99cf-6fde93ef4782" (UID: "44b27ab1-a39e-4400-99cf-6fde93ef4782"). InnerVolumeSpecName "kube-api-access-pl89r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:55:51.652846 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:51.652817    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b27ab1-a39e-4400-99cf-6fde93ef4782-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44b27ab1-a39e-4400-99cf-6fde93ef4782" (UID: "44b27ab1-a39e-4400-99cf-6fde93ef4782"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:51.751715 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:51.751685    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b27ab1-a39e-4400-99cf-6fde93ef4782-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:55:51.751715 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:51.751709    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl89r\" (UniqueName: \"kubernetes.io/projected/44b27ab1-a39e-4400-99cf-6fde93ef4782-kube-api-access-pl89r\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:55:52.096480 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:52.096387    2559 generic.go:356] "Generic (PLEG): container finished" podID="44b27ab1-a39e-4400-99cf-6fde93ef4782" containerID="19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da" exitCode=0
Jul 28 17:55:52.096480 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:52.096420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk" event={"ID":"44b27ab1-a39e-4400-99cf-6fde93ef4782","Type":"ContainerDied","Data":"19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da"}
Jul 28 17:55:52.096480 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:52.096432    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"
Jul 28 17:55:52.096480 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:52.096472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk" event={"ID":"44b27ab1-a39e-4400-99cf-6fde93ef4782","Type":"ContainerDied","Data":"b764fc32185bea7b35f197c1033c87c380b319d417c37b6d2eaf92020079478b"}
Jul 28 17:55:52.096759 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:52.096495    2559 scope.go:117] "RemoveContainer" containerID="19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da"
Jul 28 17:55:52.104684 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:52.104668    2559 scope.go:117] "RemoveContainer" containerID="19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da"
Jul 28 17:55:52.104969 ip-10-0-133-142 kubenswrapper[2559]: E0728 17:55:52.104943    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da\": container with ID starting with 19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da not found: ID does not exist" containerID="19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da"
Jul 28 17:55:52.105074 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:52.104978    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da"} err="failed to get container status \"19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da\": rpc error: code = NotFound desc = could not find container \"19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da\": container with ID starting with 19b9b7290f13490c27dda2de60eee1d039e315d78234f4ca72bb2c56e236f5da not found: ID does not exist"
Jul 28 17:55:52.136805 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:52.136776    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"]
Jul 28 17:55:52.144964 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:52.144945    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche2shbk"]
Jul 28 17:55:53.832668 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:53.832632    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b27ab1-a39e-4400-99cf-6fde93ef4782" path="/var/lib/kubelet/pods/44b27ab1-a39e-4400-99cf-6fde93ef4782/volumes"
Jul 28 17:55:56.242073 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:56.242040    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:56.346286 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:55:56.346250    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:03.061957 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:03.061913    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:04.807320 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:04.807282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:05.045847 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.045816    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"]
Jul 28 17:56:05.046153 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.046140    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b27ab1-a39e-4400-99cf-6fde93ef4782" containerName="main"
Jul 28 17:56:05.046201 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.046155    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b27ab1-a39e-4400-99cf-6fde93ef4782" containerName="main"
Jul 28 17:56:05.046236 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.046226    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b27ab1-a39e-4400-99cf-6fde93ef4782" containerName="main"
Jul 28 17:56:05.050202 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.050185    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:05.052842 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.052819    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 17:56:05.053645 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.053625    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 17:56:05.053759 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.053625    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-z284l\""
Jul 28 17:56:05.053759 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.053625    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-4xtmt\""
Jul 28 17:56:05.053759 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.053664    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 28 17:56:05.061398 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.061340    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"]
Jul 28 17:56:05.110853 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.110812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:05.167063 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.167033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05bd54d-18a9-473a-b605-fe0d54de4d8b-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw\" (UID: \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:05.167244 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.167081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spht5\" (UniqueName: \"kubernetes.io/projected/e05bd54d-18a9-473a-b605-fe0d54de4d8b-kube-api-access-spht5\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw\" (UID: \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:05.268512 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.268471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spht5\" (UniqueName: \"kubernetes.io/projected/e05bd54d-18a9-473a-b605-fe0d54de4d8b-kube-api-access-spht5\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw\" (UID: \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:05.268695 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.268579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05bd54d-18a9-473a-b605-fe0d54de4d8b-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw\" (UID: \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:05.271167 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.271143    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05bd54d-18a9-473a-b605-fe0d54de4d8b-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw\" (UID: \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:05.278568 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.278546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spht5\" (UniqueName: \"kubernetes.io/projected/e05bd54d-18a9-473a-b605-fe0d54de4d8b-kube-api-access-spht5\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw\" (UID: \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:05.361307 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.361210    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:05.504595 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:05.504541    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"]
Jul 28 17:56:05.507400 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:56:05.507365    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05bd54d_18a9_473a_b605_fe0d54de4d8b.slice/crio-42bd8b8ec43cbdad2343e5ab155dfc9c23dd500b342217e34b508b2e8e445c58 WatchSource:0}: Error finding container 42bd8b8ec43cbdad2343e5ab155dfc9c23dd500b342217e34b508b2e8e445c58: Status 404 returned error can't find the container with id 42bd8b8ec43cbdad2343e5ab155dfc9c23dd500b342217e34b508b2e8e445c58
Jul 28 17:56:06.158144 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:06.158100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw" event={"ID":"e05bd54d-18a9-473a-b605-fe0d54de4d8b","Type":"ContainerStarted","Data":"6d002f0c7a05a07d4a071f49da307d93cdaaf8889516a241cb762523065517cc"}
Jul 28 17:56:06.158617 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:06.158167    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:06.158617 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:06.158194    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw" event={"ID":"e05bd54d-18a9-473a-b605-fe0d54de4d8b","Type":"ContainerStarted","Data":"42bd8b8ec43cbdad2343e5ab155dfc9c23dd500b342217e34b508b2e8e445c58"}
Jul 28 17:56:06.197075 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:06.197026    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw" podStartSLOduration=1.197009657 podStartE2EDuration="1.197009657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:56:06.196724615 +0000 UTC m=+2006.851219626" watchObservedRunningTime="2026-07-28 17:56:06.197009657 +0000 UTC m=+2006.851504631"
Jul 28 17:56:10.588146 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:10.588102    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:10.692127 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:10.692079    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:15.733307 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:15.733268    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:37.165402 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:37.165361    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:56:56.849352 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:56.849313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:57.066334 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:56:57.066295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:09.727332 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:57:09.727238    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:31.162081 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:57:31.162040    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:31.269968 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:57:31.269928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:46.506167 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:57:46.506122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:51.532633 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:57:51.532600    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"]
Jul 28 17:57:51.537897 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:57:51.537852    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:51.631930 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:57:51.631869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:57.164707 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:57:57.164660    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw" podUID="e05bd54d-18a9-473a-b605-fe0d54de4d8b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:58:06.552693 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:06.552639    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw" podUID="e05bd54d-18a9-473a-b605-fe0d54de4d8b" containerName="main" containerID="cri-o://6d002f0c7a05a07d4a071f49da307d93cdaaf8889516a241cb762523065517cc" gracePeriod=45
Jul 28 17:58:06.680394 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:06.680358    2559 generic.go:356] "Generic (PLEG): container finished" podID="e05bd54d-18a9-473a-b605-fe0d54de4d8b" containerID="6d002f0c7a05a07d4a071f49da307d93cdaaf8889516a241cb762523065517cc" exitCode=0
Jul 28 17:58:06.680518 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:06.680426    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw" event={"ID":"e05bd54d-18a9-473a-b605-fe0d54de4d8b","Type":"ContainerDied","Data":"6d002f0c7a05a07d4a071f49da307d93cdaaf8889516a241cb762523065517cc"}
Jul 28 17:58:06.807123 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:06.807066    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:58:06.971388 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:06.971347    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05bd54d-18a9-473a-b605-fe0d54de4d8b-tls-certs\") pod \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\" (UID: \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\") "
Jul 28 17:58:06.971595 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:06.971457    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spht5\" (UniqueName: \"kubernetes.io/projected/e05bd54d-18a9-473a-b605-fe0d54de4d8b-kube-api-access-spht5\") pod \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\" (UID: \"e05bd54d-18a9-473a-b605-fe0d54de4d8b\") "
Jul 28 17:58:06.973801 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:06.973774    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05bd54d-18a9-473a-b605-fe0d54de4d8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e05bd54d-18a9-473a-b605-fe0d54de4d8b" (UID: "e05bd54d-18a9-473a-b605-fe0d54de4d8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:06.973919 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:06.973813    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05bd54d-18a9-473a-b605-fe0d54de4d8b-kube-api-access-spht5" (OuterVolumeSpecName: "kube-api-access-spht5") pod "e05bd54d-18a9-473a-b605-fe0d54de4d8b" (UID: "e05bd54d-18a9-473a-b605-fe0d54de4d8b"). InnerVolumeSpecName "kube-api-access-spht5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:58:07.072709 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:07.072623    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spht5\" (UniqueName: \"kubernetes.io/projected/e05bd54d-18a9-473a-b605-fe0d54de4d8b-kube-api-access-spht5\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:58:07.072709 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:07.072655    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05bd54d-18a9-473a-b605-fe0d54de4d8b-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 17:58:07.685920 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:07.685868    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw" event={"ID":"e05bd54d-18a9-473a-b605-fe0d54de4d8b","Type":"ContainerDied","Data":"42bd8b8ec43cbdad2343e5ab155dfc9c23dd500b342217e34b508b2e8e445c58"}
Jul 28 17:58:07.686366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:07.685946    2559 scope.go:117] "RemoveContainer" containerID="6d002f0c7a05a07d4a071f49da307d93cdaaf8889516a241cb762523065517cc"
Jul 28 17:58:07.686366 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:07.685894    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"
Jul 28 17:58:07.709587 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:07.709554    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"]
Jul 28 17:58:07.713493 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:07.713467    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewtqnw"]
Jul 28 17:58:07.834418 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:07.834386    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e05bd54d-18a9-473a-b605-fe0d54de4d8b" path="/var/lib/kubelet/pods/e05bd54d-18a9-473a-b605-fe0d54de4d8b/volumes"
Jul 28 17:58:11.363057 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:11.363025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:11.484357 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:11.484326    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:18.449034 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.448998    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:18.612738 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.612694    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:18.858080 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.858040    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"]
Jul 28 17:58:18.860301 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.858733    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e05bd54d-18a9-473a-b605-fe0d54de4d8b" containerName="main"
Jul 28 17:58:18.860301 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.858755    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e05bd54d-18a9-473a-b605-fe0d54de4d8b" containerName="main"
Jul 28 17:58:18.860301 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.858931    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e05bd54d-18a9-473a-b605-fe0d54de4d8b" containerName="main"
Jul 28 17:58:18.862860 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.862833    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:58:18.868619 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.868594    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 28 17:58:18.868837 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.868820    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-r9kbj\""
Jul 28 17:58:18.871083 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.871062    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-9nw28\""
Jul 28 17:58:18.872290 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.872275    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 17:58:18.872484 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.872463    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 17:58:18.880505 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.880482    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"]
Jul 28 17:58:18.906390 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.906351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:18.977135 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.977102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvsr\" (UniqueName: \"kubernetes.io/projected/8ecfd0c7-723a-4405-bc38-c17335f356ee-kube-api-access-gxvsr\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9\" (UID: \"8ecfd0c7-723a-4405-bc38-c17335f356ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:58:18.977135 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:18.977145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecfd0c7-723a-4405-bc38-c17335f356ee-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9\" (UID: \"8ecfd0c7-723a-4405-bc38-c17335f356ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:58:19.077781 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.077683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvsr\" (UniqueName: \"kubernetes.io/projected/8ecfd0c7-723a-4405-bc38-c17335f356ee-kube-api-access-gxvsr\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9\" (UID: \"8ecfd0c7-723a-4405-bc38-c17335f356ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:58:19.077781 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.077733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecfd0c7-723a-4405-bc38-c17335f356ee-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9\" (UID: \"8ecfd0c7-723a-4405-bc38-c17335f356ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:58:19.080343 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.080314    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecfd0c7-723a-4405-bc38-c17335f356ee-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9\" (UID: \"8ecfd0c7-723a-4405-bc38-c17335f356ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:58:19.091116 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.091085    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvsr\" (UniqueName: \"kubernetes.io/projected/8ecfd0c7-723a-4405-bc38-c17335f356ee-kube-api-access-gxvsr\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9\" (UID: \"8ecfd0c7-723a-4405-bc38-c17335f356ee\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:58:19.173076 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.173038    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:58:19.309904 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.309846    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"]
Jul 28 17:58:19.313891 ip-10-0-133-142 kubenswrapper[2559]: W0728 17:58:19.313845    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecfd0c7_723a_4405_bc38_c17335f356ee.slice/crio-3ac68667ff0bf122cbd1a091634e0bfee16f0fae2fc320f14113993fd19e6ca6 WatchSource:0}: Error finding container 3ac68667ff0bf122cbd1a091634e0bfee16f0fae2fc320f14113993fd19e6ca6: Status 404 returned error can't find the container with id 3ac68667ff0bf122cbd1a091634e0bfee16f0fae2fc320f14113993fd19e6ca6
Jul 28 17:58:19.740523 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.740487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9" event={"ID":"8ecfd0c7-723a-4405-bc38-c17335f356ee","Type":"ContainerStarted","Data":"132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719"}
Jul 28 17:58:19.741098 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.740543    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9" event={"ID":"8ecfd0c7-723a-4405-bc38-c17335f356ee","Type":"ContainerStarted","Data":"3ac68667ff0bf122cbd1a091634e0bfee16f0fae2fc320f14113993fd19e6ca6"}
Jul 28 17:58:19.741098 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.740592    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:58:19.765811 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:19.765756    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9" podStartSLOduration=1.765737076 podStartE2EDuration="1.765737076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:19.7622095 +0000 UTC m=+2140.416704477" watchObservedRunningTime="2026-07-28 17:58:19.765737076 +0000 UTC m=+2140.420232052"
Jul 28 17:58:24.746747 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:24.746706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:24.844907 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:24.844848    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:27.893074 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:27.893024    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:40.011894 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:40.011842    2559 scope.go:117] "RemoveContainer" containerID="abbc66910dcc90b3a4a4bfcfead646ba44c1b62e1725f0d43ea0b4aa975d6bac"
Jul 28 17:58:50.747132 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:58:50.747102    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 17:59:00.942750 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:00.942706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:01.158352 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:01.158288    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:06.823663 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:06.823618    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:31.472385 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:31.472343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:34.360092 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:34.360051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:34.436870 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:34.436832    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:43.665359 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:43.665317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:43.766155 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:43.766120    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:54.506242 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:54.506206    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"]
Jul 28 17:59:54.511608 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:54.511575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:54.602621 ip-10-0-133-142 kubenswrapper[2559]: I0728 17:59:54.602581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:00.301412 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:00.301372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:00.401043 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:00.401009    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:00.746217 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:00.746187    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9" podUID="8ecfd0c7-723a-4405-bc38-c17335f356ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:00:09.526168 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:09.526123    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9" podUID="8ecfd0c7-723a-4405-bc38-c17335f356ee" containerName="main" containerID="cri-o://132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719" gracePeriod=45
Jul 28 18:00:09.773789 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:09.773758    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 18:00:09.816639 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:09.816560    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxvsr\" (UniqueName: \"kubernetes.io/projected/8ecfd0c7-723a-4405-bc38-c17335f356ee-kube-api-access-gxvsr\") pod \"8ecfd0c7-723a-4405-bc38-c17335f356ee\" (UID: \"8ecfd0c7-723a-4405-bc38-c17335f356ee\") "
Jul 28 18:00:09.816639 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:09.816620    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecfd0c7-723a-4405-bc38-c17335f356ee-tls-certs\") pod \"8ecfd0c7-723a-4405-bc38-c17335f356ee\" (UID: \"8ecfd0c7-723a-4405-bc38-c17335f356ee\") "
Jul 28 18:00:09.819010 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:09.818985    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecfd0c7-723a-4405-bc38-c17335f356ee-kube-api-access-gxvsr" (OuterVolumeSpecName: "kube-api-access-gxvsr") pod "8ecfd0c7-723a-4405-bc38-c17335f356ee" (UID: "8ecfd0c7-723a-4405-bc38-c17335f356ee"). InnerVolumeSpecName "kube-api-access-gxvsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:09.819111 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:09.819082    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecfd0c7-723a-4405-bc38-c17335f356ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ecfd0c7-723a-4405-bc38-c17335f356ee" (UID: "8ecfd0c7-723a-4405-bc38-c17335f356ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:00:09.917316 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:09.917270    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecfd0c7-723a-4405-bc38-c17335f356ee-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:00:09.917316 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:09.917307    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxvsr\" (UniqueName: \"kubernetes.io/projected/8ecfd0c7-723a-4405-bc38-c17335f356ee-kube-api-access-gxvsr\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:00:10.213632 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:10.213593    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ecfd0c7-723a-4405-bc38-c17335f356ee" containerID="132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719" exitCode=0
Jul 28 18:00:10.213838 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:10.213638    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"
Jul 28 18:00:10.213838 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:10.213685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9" event={"ID":"8ecfd0c7-723a-4405-bc38-c17335f356ee","Type":"ContainerDied","Data":"132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719"}
Jul 28 18:00:10.213838 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:10.213743    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9" event={"ID":"8ecfd0c7-723a-4405-bc38-c17335f356ee","Type":"ContainerDied","Data":"3ac68667ff0bf122cbd1a091634e0bfee16f0fae2fc320f14113993fd19e6ca6"}
Jul 28 18:00:10.213838 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:10.213773    2559 scope.go:117] "RemoveContainer" containerID="132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719"
Jul 28 18:00:10.222589 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:10.222566    2559 scope.go:117] "RemoveContainer" containerID="132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719"
Jul 28 18:00:10.222855 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:00:10.222833    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719\": container with ID starting with 132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719 not found: ID does not exist" containerID="132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719"
Jul 28 18:00:10.222928 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:10.222866    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719"} err="failed to get container status \"132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719\": rpc error: code = NotFound desc = could not find container \"132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719\": container with ID starting with 132b283b86ae2e19116fa1c2f8ed520b708248a51120f3b9ef2d1b11de8d6719 not found: ID does not exist"
Jul 28 18:00:10.231265 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:10.231238    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"]
Jul 28 18:00:10.235452 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:10.235416    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sched2kh9"]
Jul 28 18:00:11.837814 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:11.837778    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ecfd0c7-723a-4405-bc38-c17335f356ee" path="/var/lib/kubelet/pods/8ecfd0c7-723a-4405-bc38-c17335f356ee/volumes"
Jul 28 18:00:14.565031 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:14.564994    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:14.662148 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:14.662118    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:22.809393 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:22.809361    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:24.323802 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.323759    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:24.631384 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.631352    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"]
Jul 28 18:00:24.631687 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.631674    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecfd0c7-723a-4405-bc38-c17335f356ee" containerName="main"
Jul 28 18:00:24.631734 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.631689    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecfd0c7-723a-4405-bc38-c17335f356ee" containerName="main"
Jul 28 18:00:24.631773 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.631748    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecfd0c7-723a-4405-bc38-c17335f356ee" containerName="main"
Jul 28 18:00:24.634644 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.634625    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:00:24.640735 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.640712    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-wng8v\""
Jul 28 18:00:24.641224 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.640731    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 28 18:00:24.641224 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.640754    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 28 18:00:24.641352 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.641314    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-q96q9\""
Jul 28 18:00:24.646559 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.646537    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 28 18:00:24.662155 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.662135    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"]
Jul 28 18:00:24.670349 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.670324    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:24.738015 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.737982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhx6\" (UniqueName: \"kubernetes.io/projected/81c4b53e-cad7-44c3-bd84-d522157fa3cd-kube-api-access-gvhx6\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh\" (UID: \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:00:24.738200 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.738037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c4b53e-cad7-44c3-bd84-d522157fa3cd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh\" (UID: \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:00:24.839082 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.839041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhx6\" (UniqueName: \"kubernetes.io/projected/81c4b53e-cad7-44c3-bd84-d522157fa3cd-kube-api-access-gvhx6\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh\" (UID: \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:00:24.839261 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.839104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c4b53e-cad7-44c3-bd84-d522157fa3cd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh\" (UID: \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:00:24.841620 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.841590    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c4b53e-cad7-44c3-bd84-d522157fa3cd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh\" (UID: \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:00:24.849625 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.849599    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhx6\" (UniqueName: \"kubernetes.io/projected/81c4b53e-cad7-44c3-bd84-d522157fa3cd-kube-api-access-gvhx6\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh\" (UID: \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:00:24.944697 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:24.944613    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:00:25.080478 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:25.080425    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"]
Jul 28 18:00:25.082782 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:00:25.082745    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c4b53e_cad7_44c3_bd84_d522157fa3cd.slice/crio-8f94e9c941ceed2936fd0d0ab3517bc937de1b137314327a8f27f10d406ea875 WatchSource:0}: Error finding container 8f94e9c941ceed2936fd0d0ab3517bc937de1b137314327a8f27f10d406ea875: Status 404 returned error can't find the container with id 8f94e9c941ceed2936fd0d0ab3517bc937de1b137314327a8f27f10d406ea875
Jul 28 18:00:25.289579 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:25.289463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh" event={"ID":"81c4b53e-cad7-44c3-bd84-d522157fa3cd","Type":"ContainerStarted","Data":"6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9"}
Jul 28 18:00:25.289738 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:25.289600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh" event={"ID":"81c4b53e-cad7-44c3-bd84-d522157fa3cd","Type":"ContainerStarted","Data":"8f94e9c941ceed2936fd0d0ab3517bc937de1b137314327a8f27f10d406ea875"}
Jul 28 18:00:25.289738 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:25.289634    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:00:25.322951 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:25.322893    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh" podStartSLOduration=1.322860235 podStartE2EDuration="1.322860235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:25.321764288 +0000 UTC m=+2265.976259263" watchObservedRunningTime="2026-07-28 18:00:25.322860235 +0000 UTC m=+2265.977355246"
Jul 28 18:00:29.861391 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:29.861356    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:29.969543 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:29.969500    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:36.594159 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:36.594123    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:56.295781 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:00:56.295750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:02:20.199464 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:20.199428    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"]
Jul 28 18:02:20.211743 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:20.211714    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:20.316965 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:20.316927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:26.296656 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:26.296615    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh" podUID="81c4b53e-cad7-44c3-bd84-d522157fa3cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:02:35.218858 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.218814    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh" podUID="81c4b53e-cad7-44c3-bd84-d522157fa3cd" containerName="main" containerID="cri-o://6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9" gracePeriod=45
Jul 28 18:02:35.466309 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.466281    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:02:35.550950 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.550841    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c4b53e-cad7-44c3-bd84-d522157fa3cd-tls-certs\") pod \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\" (UID: \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\") "
Jul 28 18:02:35.550950 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.550932    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvhx6\" (UniqueName: \"kubernetes.io/projected/81c4b53e-cad7-44c3-bd84-d522157fa3cd-kube-api-access-gvhx6\") pod \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\" (UID: \"81c4b53e-cad7-44c3-bd84-d522157fa3cd\") "
Jul 28 18:02:35.553253 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.553220    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c4b53e-cad7-44c3-bd84-d522157fa3cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81c4b53e-cad7-44c3-bd84-d522157fa3cd" (UID: "81c4b53e-cad7-44c3-bd84-d522157fa3cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:02:35.553373 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.553226    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c4b53e-cad7-44c3-bd84-d522157fa3cd-kube-api-access-gvhx6" (OuterVolumeSpecName: "kube-api-access-gvhx6") pod "81c4b53e-cad7-44c3-bd84-d522157fa3cd" (UID: "81c4b53e-cad7-44c3-bd84-d522157fa3cd"). InnerVolumeSpecName "kube-api-access-gvhx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:02:35.652339 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.652303    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c4b53e-cad7-44c3-bd84-d522157fa3cd-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:02:35.652339 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.652332    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhx6\" (UniqueName: \"kubernetes.io/projected/81c4b53e-cad7-44c3-bd84-d522157fa3cd-kube-api-access-gvhx6\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:02:35.868525 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.868492    2559 generic.go:356] "Generic (PLEG): container finished" podID="81c4b53e-cad7-44c3-bd84-d522157fa3cd" containerID="6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9" exitCode=0
Jul 28 18:02:35.868693 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.868541    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"
Jul 28 18:02:35.868693 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.868576    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh" event={"ID":"81c4b53e-cad7-44c3-bd84-d522157fa3cd","Type":"ContainerDied","Data":"6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9"}
Jul 28 18:02:35.868693 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.868624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh" event={"ID":"81c4b53e-cad7-44c3-bd84-d522157fa3cd","Type":"ContainerDied","Data":"8f94e9c941ceed2936fd0d0ab3517bc937de1b137314327a8f27f10d406ea875"}
Jul 28 18:02:35.868693 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.868648    2559 scope.go:117] "RemoveContainer" containerID="6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9"
Jul 28 18:02:35.876905 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.876865    2559 scope.go:117] "RemoveContainer" containerID="6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9"
Jul 28 18:02:35.877156 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:02:35.877137    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9\": container with ID starting with 6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9 not found: ID does not exist" containerID="6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9"
Jul 28 18:02:35.877198 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.877167    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9"} err="failed to get container status \"6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9\": rpc error: code = NotFound desc = could not find container \"6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9\": container with ID starting with 6598059cdc578204a8622662cf94bfab3c0b76e106589d4832b19a8c082166d9 not found: ID does not exist"
Jul 28 18:02:35.885391 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.885361    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"]
Jul 28 18:02:35.891432 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:35.891413    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche26xxh"]
Jul 28 18:02:37.833038 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:37.833003    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c4b53e-cad7-44c3-bd84-d522157fa3cd" path="/var/lib/kubelet/pods/81c4b53e-cad7-44c3-bd84-d522157fa3cd/volumes"
Jul 28 18:02:40.132870 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:40.132835    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:40.222707 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:40.222668    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:48.259242 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:48.259181    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:51.950168 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:51.950130    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:52.372194 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.372158    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"]
Jul 28 18:02:52.372498 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.372485    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c4b53e-cad7-44c3-bd84-d522157fa3cd" containerName="main"
Jul 28 18:02:52.372555 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.372500    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c4b53e-cad7-44c3-bd84-d522157fa3cd" containerName="main"
Jul 28 18:02:52.372593 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.372569    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c4b53e-cad7-44c3-bd84-d522157fa3cd" containerName="main"
Jul 28 18:02:52.375439 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.375421    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:02:52.381484 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.381461    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-29cg4\""
Jul 28 18:02:52.381639 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.381483    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 28 18:02:52.381639 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.381462    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 28 18:02:52.381639 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.381485    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-nxtkr\""
Jul 28 18:02:52.381639 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.381462    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 28 18:02:52.392746 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.392725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gspbf\" (UniqueName: \"kubernetes.io/projected/f25e55d8-ead8-4d97-9555-253bebc3a180-kube-api-access-gspbf\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt\" (UID: \"f25e55d8-ead8-4d97-9555-253bebc3a180\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:02:52.392846 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.392787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f25e55d8-ead8-4d97-9555-253bebc3a180-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt\" (UID: \"f25e55d8-ead8-4d97-9555-253bebc3a180\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:02:52.405961 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.405934    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"]
Jul 28 18:02:52.415041 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.415015    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:52.493515 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.493480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f25e55d8-ead8-4d97-9555-253bebc3a180-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt\" (UID: \"f25e55d8-ead8-4d97-9555-253bebc3a180\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:02:52.493692 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.493538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gspbf\" (UniqueName: \"kubernetes.io/projected/f25e55d8-ead8-4d97-9555-253bebc3a180-kube-api-access-gspbf\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt\" (UID: \"f25e55d8-ead8-4d97-9555-253bebc3a180\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:02:52.496338 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.496309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f25e55d8-ead8-4d97-9555-253bebc3a180-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt\" (UID: \"f25e55d8-ead8-4d97-9555-253bebc3a180\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:02:52.506185 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.506162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gspbf\" (UniqueName: \"kubernetes.io/projected/f25e55d8-ead8-4d97-9555-253bebc3a180-kube-api-access-gspbf\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt\" (UID: \"f25e55d8-ead8-4d97-9555-253bebc3a180\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:02:52.686363 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.686273    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:02:52.840988 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.840956    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"]
Jul 28 18:02:52.844024 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:02:52.843997    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25e55d8_ead8_4d97_9555_253bebc3a180.slice/crio-0aafe51199ac11336b1f89e2fab1ef3e8d6d415beb724234f80f96d49b888f46 WatchSource:0}: Error finding container 0aafe51199ac11336b1f89e2fab1ef3e8d6d415beb724234f80f96d49b888f46: Status 404 returned error can't find the container with id 0aafe51199ac11336b1f89e2fab1ef3e8d6d415beb724234f80f96d49b888f46
Jul 28 18:02:52.947149 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.947044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt" event={"ID":"f25e55d8-ead8-4d97-9555-253bebc3a180","Type":"ContainerStarted","Data":"8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3"}
Jul 28 18:02:52.947149 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.947116    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt" event={"ID":"f25e55d8-ead8-4d97-9555-253bebc3a180","Type":"ContainerStarted","Data":"0aafe51199ac11336b1f89e2fab1ef3e8d6d415beb724234f80f96d49b888f46"}
Jul 28 18:02:52.947149 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.947147    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:02:52.995331 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:52.995273    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt" podStartSLOduration=0.995256486 podStartE2EDuration="995.256486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:52.990714798 +0000 UTC m=+2413.645209864" watchObservedRunningTime="2026-07-28 18:02:52.995256486 +0000 UTC m=+2413.649751461"
Jul 28 18:02:56.208175 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:56.208140    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:56.299234 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:56.299194    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:59.394141 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:02:59.394108    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:12.902753 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:12.902546    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:12.909060 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:12.909030    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"]
Jul 28 18:03:13.174062 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:13.173968    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq" podUID="1edb2f85-d702-421f-bf86-96324a03c935" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:03:13.765064 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:13.765025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:23.174070 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:23.174027    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq" podUID="1edb2f85-d702-421f-bf86-96324a03c935" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:03:23.953612 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:23.953582    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:03:27.930505 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:27.930443    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq" podUID="1edb2f85-d702-421f-bf86-96324a03c935" containerName="main" containerID="cri-o://1ad5188e441d2aca48bd27ca4b6cbd457401078f38981a828c6e876c90b51b8b" gracePeriod=45
Jul 28 18:03:28.123953 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:28.123912    2559 generic.go:356] "Generic (PLEG): container finished" podID="1edb2f85-d702-421f-bf86-96324a03c935" containerID="1ad5188e441d2aca48bd27ca4b6cbd457401078f38981a828c6e876c90b51b8b" exitCode=0
Jul 28 18:03:28.124138 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:28.124045    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq" event={"ID":"1edb2f85-d702-421f-bf86-96324a03c935","Type":"ContainerDied","Data":"1ad5188e441d2aca48bd27ca4b6cbd457401078f38981a828c6e876c90b51b8b"}
Jul 28 18:03:28.191678 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:28.191614    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 18:03:28.295629 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:28.295595    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb2f85-d702-421f-bf86-96324a03c935-tls-certs\") pod \"1edb2f85-d702-421f-bf86-96324a03c935\" (UID: \"1edb2f85-d702-421f-bf86-96324a03c935\") "
Jul 28 18:03:28.295797 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:28.295643    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sndc\" (UniqueName: \"kubernetes.io/projected/1edb2f85-d702-421f-bf86-96324a03c935-kube-api-access-9sndc\") pod \"1edb2f85-d702-421f-bf86-96324a03c935\" (UID: \"1edb2f85-d702-421f-bf86-96324a03c935\") "
Jul 28 18:03:28.297993 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:28.297960    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1edb2f85-d702-421f-bf86-96324a03c935-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1edb2f85-d702-421f-bf86-96324a03c935" (UID: "1edb2f85-d702-421f-bf86-96324a03c935"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:03:28.297993 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:28.297988    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edb2f85-d702-421f-bf86-96324a03c935-kube-api-access-9sndc" (OuterVolumeSpecName: "kube-api-access-9sndc") pod "1edb2f85-d702-421f-bf86-96324a03c935" (UID: "1edb2f85-d702-421f-bf86-96324a03c935"). InnerVolumeSpecName "kube-api-access-9sndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:03:28.396690 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:28.396657    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edb2f85-d702-421f-bf86-96324a03c935-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:03:28.396690 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:28.396685    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sndc\" (UniqueName: \"kubernetes.io/projected/1edb2f85-d702-421f-bf86-96324a03c935-kube-api-access-9sndc\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:03:29.129777 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:29.129737    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"
Jul 28 18:03:29.129777 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:29.129752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq" event={"ID":"1edb2f85-d702-421f-bf86-96324a03c935","Type":"ContainerDied","Data":"0befea57b3f519c5d83e40d74b17fddc853e1b00311bb17166385eafe557bf3a"}
Jul 28 18:03:29.130304 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:29.129815    2559 scope.go:117] "RemoveContainer" containerID="1ad5188e441d2aca48bd27ca4b6cbd457401078f38981a828c6e876c90b51b8b"
Jul 28 18:03:29.157789 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:29.157764    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"]
Jul 28 18:03:29.162509 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:29.162486    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq"]
Jul 28 18:03:29.833598 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:29.833568    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1edb2f85-d702-421f-bf86-96324a03c935" path="/var/lib/kubelet/pods/1edb2f85-d702-421f-bf86-96324a03c935/volumes"
Jul 28 18:03:33.132081 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:33.132046    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:33.228885 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:33.228850    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:36.436474 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:36.436434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:37.034571 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.034538    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"]
Jul 28 18:03:37.034926 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.034910    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1edb2f85-d702-421f-bf86-96324a03c935" containerName="main"
Jul 28 18:03:37.034926 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.034926    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1edb2f85-d702-421f-bf86-96324a03c935" containerName="main"
Jul 28 18:03:37.035057 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.034996    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1edb2f85-d702-421f-bf86-96324a03c935" containerName="main"
Jul 28 18:03:37.039227 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.039209    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:03:37.043084 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.043059    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 28 18:03:37.043084 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.043086    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 28 18:03:37.043661 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.043638    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-mq9rw\""
Jul 28 18:03:37.043661 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.043644    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-epp-sa-dockercfg-6xqf7\""
Jul 28 18:03:37.043832 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.043675    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 28 18:03:37.053260 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.053238    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"]
Jul 28 18:03:37.081232 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.081200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:37.172218 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.172185    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842bc59d-e343-4d92-9535-dcbfbe456e56-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57\" (UID: \"842bc59d-e343-4d92-9535-dcbfbe456e56\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:03:37.172218 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.172218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq6fl\" (UniqueName: \"kubernetes.io/projected/842bc59d-e343-4d92-9535-dcbfbe456e56-kube-api-access-cq6fl\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57\" (UID: \"842bc59d-e343-4d92-9535-dcbfbe456e56\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:03:37.272907 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.272841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842bc59d-e343-4d92-9535-dcbfbe456e56-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57\" (UID: \"842bc59d-e343-4d92-9535-dcbfbe456e56\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:03:37.272907 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.272907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq6fl\" (UniqueName: \"kubernetes.io/projected/842bc59d-e343-4d92-9535-dcbfbe456e56-kube-api-access-cq6fl\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57\" (UID: \"842bc59d-e343-4d92-9535-dcbfbe456e56\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:03:37.275426 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.275403    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842bc59d-e343-4d92-9535-dcbfbe456e56-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57\" (UID: \"842bc59d-e343-4d92-9535-dcbfbe456e56\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:03:37.283262 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.283231    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq6fl\" (UniqueName: \"kubernetes.io/projected/842bc59d-e343-4d92-9535-dcbfbe456e56-kube-api-access-cq6fl\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57\" (UID: \"842bc59d-e343-4d92-9535-dcbfbe456e56\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:03:37.350407 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.350310    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:03:37.494737 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:37.494684    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"]
Jul 28 18:03:37.497649 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:03:37.497624    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842bc59d_e343_4d92_9535_dcbfbe456e56.slice/crio-a6497adc9eaecc070e465ed67c77daf8ae59ca10a2bb0c2db45db4d3a6230304 WatchSource:0}: Error finding container a6497adc9eaecc070e465ed67c77daf8ae59ca10a2bb0c2db45db4d3a6230304: Status 404 returned error can't find the container with id a6497adc9eaecc070e465ed67c77daf8ae59ca10a2bb0c2db45db4d3a6230304
Jul 28 18:03:38.174192 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:38.174153    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57" event={"ID":"842bc59d-e343-4d92-9535-dcbfbe456e56","Type":"ContainerStarted","Data":"a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7"}
Jul 28 18:03:38.174192 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:38.174198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57" event={"ID":"842bc59d-e343-4d92-9535-dcbfbe456e56","Type":"ContainerStarted","Data":"a6497adc9eaecc070e465ed67c77daf8ae59ca10a2bb0c2db45db4d3a6230304"}
Jul 28 18:03:38.174472 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:38.174284    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:03:38.212568 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:38.212508    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57" podStartSLOduration=1.212493479 podStartE2EDuration="1.212493479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:38.21147106 +0000 UTC m=+2458.865966053" watchObservedRunningTime="2026-07-28 18:03:38.212493479 +0000 UTC m=+2458.866988454"
Jul 28 18:03:42.720150 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:42.720110    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:58.788241 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:03:58.788199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:09.179953 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:09.179924    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:04:11.916349 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:11.916310    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:24.609224 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:24.609187    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"]
Jul 28 18:04:24.658814 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:24.658780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:24.734565 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:24.732771    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:33.953620 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:33.953586    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt" podUID="f25e55d8-ead8-4d97-9555-253bebc3a180" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:04:39.628317 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:39.628213    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt" podUID="f25e55d8-ead8-4d97-9555-253bebc3a180" containerName="main" containerID="cri-o://8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3" gracePeriod=45
Jul 28 18:04:39.878563 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:39.878463    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:04:40.002944 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.002916    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f25e55d8-ead8-4d97-9555-253bebc3a180-tls-certs\") pod \"f25e55d8-ead8-4d97-9555-253bebc3a180\" (UID: \"f25e55d8-ead8-4d97-9555-253bebc3a180\") "
Jul 28 18:04:40.003110 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.002990    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gspbf\" (UniqueName: \"kubernetes.io/projected/f25e55d8-ead8-4d97-9555-253bebc3a180-kube-api-access-gspbf\") pod \"f25e55d8-ead8-4d97-9555-253bebc3a180\" (UID: \"f25e55d8-ead8-4d97-9555-253bebc3a180\") "
Jul 28 18:04:40.005210 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.005178    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25e55d8-ead8-4d97-9555-253bebc3a180-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f25e55d8-ead8-4d97-9555-253bebc3a180" (UID: "f25e55d8-ead8-4d97-9555-253bebc3a180"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:40.005320 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.005237    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25e55d8-ead8-4d97-9555-253bebc3a180-kube-api-access-gspbf" (OuterVolumeSpecName: "kube-api-access-gspbf") pod "f25e55d8-ead8-4d97-9555-253bebc3a180" (UID: "f25e55d8-ead8-4d97-9555-253bebc3a180"). InnerVolumeSpecName "kube-api-access-gspbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:40.103754 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.103724    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f25e55d8-ead8-4d97-9555-253bebc3a180-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.103754 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.103749    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gspbf\" (UniqueName: \"kubernetes.io/projected/f25e55d8-ead8-4d97-9555-253bebc3a180-kube-api-access-gspbf\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.449843 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.449802    2559 generic.go:356] "Generic (PLEG): container finished" podID="f25e55d8-ead8-4d97-9555-253bebc3a180" containerID="8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3" exitCode=0
Jul 28 18:04:40.450075 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.449863    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"
Jul 28 18:04:40.450075 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.449864    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt" event={"ID":"f25e55d8-ead8-4d97-9555-253bebc3a180","Type":"ContainerDied","Data":"8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3"}
Jul 28 18:04:40.450075 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.449959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt" event={"ID":"f25e55d8-ead8-4d97-9555-253bebc3a180","Type":"ContainerDied","Data":"0aafe51199ac11336b1f89e2fab1ef3e8d6d415beb724234f80f96d49b888f46"}
Jul 28 18:04:40.450075 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.449983    2559 scope.go:117] "RemoveContainer" containerID="8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3"
Jul 28 18:04:40.459080 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.459062    2559 scope.go:117] "RemoveContainer" containerID="8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3"
Jul 28 18:04:40.459359 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:04:40.459342    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3\": container with ID starting with 8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3 not found: ID does not exist" containerID="8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3"
Jul 28 18:04:40.459400 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.459369    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3"} err="failed to get container status \"8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3\": rpc error: code = NotFound desc = could not find container \"8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3\": container with ID starting with 8a2db6b0ad8da0ba4e2437b5ce4e48d161e7f33e59a151c0af00f3a6cfa2c4c3 not found: ID does not exist"
Jul 28 18:04:40.473487 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.473456    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"]
Jul 28 18:04:40.477662 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:40.477640    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepktnt"]
Jul 28 18:04:41.832719 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:41.832683    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f25e55d8-ead8-4d97-9555-253bebc3a180" path="/var/lib/kubelet/pods/f25e55d8-ead8-4d97-9555-253bebc3a180/volumes"
Jul 28 18:04:44.614976 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:44.614939    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:44.741980 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:44.741945    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:44.931287 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:44.931178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:45.064819 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:45.064766    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:45.167358 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:45.167308    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:47.409616 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:47.409527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:47.513079 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:47.513035    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:48.203470 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:48.203432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:51.970890 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:51.970840    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:58.251965 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:58.251930    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:58.602527 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:04:58.602450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:24.672101 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.672064    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:24.978770 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.978686    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"]
Jul 28 18:05:24.979008 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.978995    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25e55d8-ead8-4d97-9555-253bebc3a180" containerName="main"
Jul 28 18:05:24.979055 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.979010    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25e55d8-ead8-4d97-9555-253bebc3a180" containerName="main"
Jul 28 18:05:24.979091 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.979068    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25e55d8-ead8-4d97-9555-253bebc3a180" containerName="main"
Jul 28 18:05:24.981851 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.981834    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:05:24.984487 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.984465    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-r8t7m\""
Jul 28 18:05:24.985375 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.985354    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 18:05:24.985540 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.985519    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 28 18:05:24.986007 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.985988    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 18:05:24.986983 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.986968    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-6t8zk\""
Jul 28 18:05:24.994725 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.994705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"]
Jul 28 18:05:24.999567 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:24.999543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:25.051433 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.051381    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnzgz\" (UniqueName: \"kubernetes.io/projected/ebfbfe60-621d-4002-98ae-8149f75f89c7-kube-api-access-pnzgz\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs\" (UID: \"ebfbfe60-621d-4002-98ae-8149f75f89c7\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:05:25.051674 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.051489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfbfe60-621d-4002-98ae-8149f75f89c7-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs\" (UID: \"ebfbfe60-621d-4002-98ae-8149f75f89c7\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:05:25.152604 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.152566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfbfe60-621d-4002-98ae-8149f75f89c7-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs\" (UID: \"ebfbfe60-621d-4002-98ae-8149f75f89c7\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:05:25.152789 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.152632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzgz\" (UniqueName: \"kubernetes.io/projected/ebfbfe60-621d-4002-98ae-8149f75f89c7-kube-api-access-pnzgz\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs\" (UID: \"ebfbfe60-621d-4002-98ae-8149f75f89c7\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:05:25.155216 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.155194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfbfe60-621d-4002-98ae-8149f75f89c7-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs\" (UID: \"ebfbfe60-621d-4002-98ae-8149f75f89c7\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:05:25.169047 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.169022    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzgz\" (UniqueName: \"kubernetes.io/projected/ebfbfe60-621d-4002-98ae-8149f75f89c7-kube-api-access-pnzgz\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs\" (UID: \"ebfbfe60-621d-4002-98ae-8149f75f89c7\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:05:25.292941 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.292831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:05:25.434214 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.434180    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"]
Jul 28 18:05:25.437621 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:05:25.437592    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfbfe60_621d_4002_98ae_8149f75f89c7.slice/crio-a4cc6ed2728728ce88d2b01b2186fa52a75cf300bd4fb2288c8b1e5ceea2ad9a WatchSource:0}: Error finding container a4cc6ed2728728ce88d2b01b2186fa52a75cf300bd4fb2288c8b1e5ceea2ad9a: Status 404 returned error can't find the container with id a4cc6ed2728728ce88d2b01b2186fa52a75cf300bd4fb2288c8b1e5ceea2ad9a
Jul 28 18:05:25.646442 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.646405    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs" event={"ID":"ebfbfe60-621d-4002-98ae-8149f75f89c7","Type":"ContainerStarted","Data":"1e9ab5aeebae3d0dcd24d7897e338ab979d856ab2e992c86018d90ad18312350"}
Jul 28 18:05:25.646654 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.646458    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs" event={"ID":"ebfbfe60-621d-4002-98ae-8149f75f89c7","Type":"ContainerStarted","Data":"a4cc6ed2728728ce88d2b01b2186fa52a75cf300bd4fb2288c8b1e5ceea2ad9a"}
Jul 28 18:05:25.646654 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.646523    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:05:25.699335 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:25.699276    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs" podStartSLOduration=1.699257336 podStartE2EDuration="1.699257336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:05:25.696448855 +0000 UTC m=+2566.350943830" watchObservedRunningTime="2026-07-28 18:05:25.699257336 +0000 UTC m=+2566.353752311"
Jul 28 18:05:35.145929 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:35.145856    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:56.652503 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:05:56.652473    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:06:07.268629 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:06:07.268592    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:42.810688 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:06:42.810655    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:42.912046 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:06:42.912011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:48.834409 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:06:48.834370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:36.994057 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:36.994022    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:36.997355 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:36.997326    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"]
Jul 28 18:07:37.086593 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:37.086553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:46.653038 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:46.652998    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs" podUID="ebfbfe60-621d-4002-98ae-8149f75f89c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:07:52.016385 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.016319    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs" podUID="ebfbfe60-621d-4002-98ae-8149f75f89c7" containerName="main" containerID="cri-o://1e9ab5aeebae3d0dcd24d7897e338ab979d856ab2e992c86018d90ad18312350" gracePeriod=45
Jul 28 18:07:52.262639 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.262604    2559 generic.go:356] "Generic (PLEG): container finished" podID="ebfbfe60-621d-4002-98ae-8149f75f89c7" containerID="1e9ab5aeebae3d0dcd24d7897e338ab979d856ab2e992c86018d90ad18312350" exitCode=0
Jul 28 18:07:52.262771 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.262673    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs" event={"ID":"ebfbfe60-621d-4002-98ae-8149f75f89c7","Type":"ContainerDied","Data":"1e9ab5aeebae3d0dcd24d7897e338ab979d856ab2e992c86018d90ad18312350"}
Jul 28 18:07:52.262771 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.262735    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs" event={"ID":"ebfbfe60-621d-4002-98ae-8149f75f89c7","Type":"ContainerDied","Data":"a4cc6ed2728728ce88d2b01b2186fa52a75cf300bd4fb2288c8b1e5ceea2ad9a"}
Jul 28 18:07:52.262771 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.262752    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4cc6ed2728728ce88d2b01b2186fa52a75cf300bd4fb2288c8b1e5ceea2ad9a"
Jul 28 18:07:52.264856 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.264840    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:07:52.370722 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.370686    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnzgz\" (UniqueName: \"kubernetes.io/projected/ebfbfe60-621d-4002-98ae-8149f75f89c7-kube-api-access-pnzgz\") pod \"ebfbfe60-621d-4002-98ae-8149f75f89c7\" (UID: \"ebfbfe60-621d-4002-98ae-8149f75f89c7\") "
Jul 28 18:07:52.370955 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.370741    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfbfe60-621d-4002-98ae-8149f75f89c7-tls-certs\") pod \"ebfbfe60-621d-4002-98ae-8149f75f89c7\" (UID: \"ebfbfe60-621d-4002-98ae-8149f75f89c7\") "
Jul 28 18:07:52.373091 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.373061    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebfbfe60-621d-4002-98ae-8149f75f89c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebfbfe60-621d-4002-98ae-8149f75f89c7" (UID: "ebfbfe60-621d-4002-98ae-8149f75f89c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:07:52.373199 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.373114    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfbfe60-621d-4002-98ae-8149f75f89c7-kube-api-access-pnzgz" (OuterVolumeSpecName: "kube-api-access-pnzgz") pod "ebfbfe60-621d-4002-98ae-8149f75f89c7" (UID: "ebfbfe60-621d-4002-98ae-8149f75f89c7"). InnerVolumeSpecName "kube-api-access-pnzgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:07:52.471354 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.471314    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnzgz\" (UniqueName: \"kubernetes.io/projected/ebfbfe60-621d-4002-98ae-8149f75f89c7-kube-api-access-pnzgz\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:07:52.471354 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:52.471347    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebfbfe60-621d-4002-98ae-8149f75f89c7-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:07:53.266379 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:53.266350    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"
Jul 28 18:07:53.288671 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:53.288639    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"]
Jul 28 18:07:53.293275 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:53.293252    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schewhnfs"]
Jul 28 18:07:53.832788 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:53.832758    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebfbfe60-621d-4002-98ae-8149f75f89c7" path="/var/lib/kubelet/pods/ebfbfe60-621d-4002-98ae-8149f75f89c7/volumes"
Jul 28 18:07:56.945996 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:56.945956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:57.033468 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:57.033433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:57.144066 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:57.144030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:57.239982 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:57.239868    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:57.339532 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:57.339496    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:57.442614 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:57.442572    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:59.575642 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:59.575603    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:59.675328 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:07:59.675295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:00.429249 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:00.429215    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:02.577348 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:02.577315    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:02.676860 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:02.676825    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:02.918862 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:02.918824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:03.019522 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:03.019481    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:04.154176 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:04.154140    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:06.791301 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:06.791267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:10.361136 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:10.361095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:10.424900 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:10.424842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:36.456764 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.456728    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"]
Jul 28 18:08:36.457390 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.457224    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebfbfe60-621d-4002-98ae-8149f75f89c7" containerName="main"
Jul 28 18:08:36.457390 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.457243    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebfbfe60-621d-4002-98ae-8149f75f89c7" containerName="main"
Jul 28 18:08:36.457390 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.457332    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebfbfe60-621d-4002-98ae-8149f75f89c7" containerName="main"
Jul 28 18:08:36.459901 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.459859    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.463319 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.463129    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 18:08:36.463446 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.463135    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-xjjgk\""
Jul 28 18:08:36.463731 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.463582    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 18:08:36.464906 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.463989    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 18:08:36.481399 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.481380    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"]
Jul 28 18:08:36.500242 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.500210    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66n2\" (UniqueName: \"kubernetes.io/projected/e40a0463-3c22-4106-a788-24bb2ce11bfc-kube-api-access-f66n2\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.500369 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.500264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.500369 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.500346    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.500497 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.500391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.500497 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.500418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40a0463-3c22-4106-a788-24bb2ce11bfc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.500497 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.500468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.511490 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.511464    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:36.601637 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.601608    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.601757 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.601672    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66n2\" (UniqueName: \"kubernetes.io/projected/e40a0463-3c22-4106-a788-24bb2ce11bfc-kube-api-access-f66n2\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.601757 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.601723    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.601870 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.601768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.601870 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.601798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.601870 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.601819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40a0463-3c22-4106-a788-24bb2ce11bfc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.602193 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.602165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.602193 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.602185    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.602345 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.602225    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.603982 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.603966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40a0463-3c22-4106-a788-24bb2ce11bfc-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.604281 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.604265    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40a0463-3c22-4106-a788-24bb2ce11bfc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.611136 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.611112    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66n2\" (UniqueName: \"kubernetes.io/projected/e40a0463-3c22-4106-a788-24bb2ce11bfc-kube-api-access-f66n2\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:36.819097 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:36.819006    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:45.175466 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.175426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:45.181560 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.181533    2559 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:45.276086 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.276047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:45.276252 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.276158    2559 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:45.283777 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.283753    2559 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") DevicePath \"csi-79bfcbce2c303a7be4e0da631f9325c7283bc1a4d6c0ccaf8fcae15a467ca4d1\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:45.384322 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.384279    2559 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a2b1b41a2708640572ef6a0f5ba7fe5be71361ae6201f7c3e4b363026141d9c9/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:45.407135 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.407107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f\" (UID: \"e40a0463-3c22-4106-a788-24bb2ce11bfc\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:45.475062 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.474982    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-xjjgk\""
Jul 28 18:08:45.482970 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.482936    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:08:45.625420 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.625394    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"]
Jul 28 18:08:45.627180 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:08:45.627151    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40a0463_3c22_4106_a788_24bb2ce11bfc.slice/crio-4637b8dbca908fc95f42292fbe040ed53cc331a39aacbdb338b7124dcd6abc0b WatchSource:0}: Error finding container 4637b8dbca908fc95f42292fbe040ed53cc331a39aacbdb338b7124dcd6abc0b: Status 404 returned error can't find the container with id 4637b8dbca908fc95f42292fbe040ed53cc331a39aacbdb338b7124dcd6abc0b
Jul 28 18:08:45.631970 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:45.631943    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:08:46.497510 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:46.497475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" event={"ID":"e40a0463-3c22-4106-a788-24bb2ce11bfc","Type":"ContainerStarted","Data":"4637b8dbca908fc95f42292fbe040ed53cc331a39aacbdb338b7124dcd6abc0b"}
Jul 28 18:08:49.211783 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:08:49.211745    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:09:14.650354 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:09:14.650308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" event={"ID":"e40a0463-3c22-4106-a788-24bb2ce11bfc","Type":"ContainerStarted","Data":"669fcca57cec4a33eabcb6b6521c37237ac8b5d94e6fb35ad9e15d9436fe768c"}
Jul 28 18:09:14.675260 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:09:14.675202    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podStartSLOduration=9.821986105 podStartE2EDuration="38.675182861s" totalImagesPullingTime="28.853196756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:14.673166768 +0000 UTC m=+2795.327661744" watchObservedRunningTime="2026-07-28 18:09:14.675182861 +0000 UTC m=+2795.329677859"
Jul 28 18:09:15.483576 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:09:15.483543    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:09:15.483768 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:09:15.483600    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:09:15.485183 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:09:15.485152    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:09:25.483738 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:09:25.483697    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:09:35.483611 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:09:35.483567    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:09:45.483637 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:09:45.483590    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:09:55.483996 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:09:55.483950    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:10:05.484152 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:10:05.484111    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:10:15.483770 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:10:15.483717    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:10:25.483604 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:10:25.483556    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:10:35.484359 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:10:35.484312    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:10:45.484124 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:10:45.484080    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:10:55.484355 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:10:55.484302    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:11:05.484325 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:11:05.484268    2559 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f" podUID="e40a0463-3c22-4106-a788-24bb2ce11bfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 28 18:11:11.623194 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:11:11.623155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:15.493085 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:11:15.493052    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:11:15.500982 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:11:15.500956    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f"
Jul 28 18:11:40.127535 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:11:40.127504    2559 scope.go:117] "RemoveContainer" containerID="1e9ab5aeebae3d0dcd24d7897e338ab979d856ab2e992c86018d90ad18312350"
Jul 28 18:14:21.018862 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:14:21.018828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:12.010862 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:12.010779    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:24.922844 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:24.922810    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:40.089305 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:40.089270    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:40.092072 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:40.092044    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"]
Jul 28 18:18:40.195633 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:40.195596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:49.180024 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:49.179985    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57" podUID="842bc59d-e343-4d92-9535-dcbfbe456e56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:18:52.670740 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:52.670706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:55.112304 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:55.112257    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57" podUID="842bc59d-e343-4d92-9535-dcbfbe456e56" containerName="main" containerID="cri-o://a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7" gracePeriod=45
Jul 28 18:18:55.358016 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:55.357991    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:18:55.520917 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:55.520812    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq6fl\" (UniqueName: \"kubernetes.io/projected/842bc59d-e343-4d92-9535-dcbfbe456e56-kube-api-access-cq6fl\") pod \"842bc59d-e343-4d92-9535-dcbfbe456e56\" (UID: \"842bc59d-e343-4d92-9535-dcbfbe456e56\") "
Jul 28 18:18:55.520917 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:55.520901    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842bc59d-e343-4d92-9535-dcbfbe456e56-tls-certs\") pod \"842bc59d-e343-4d92-9535-dcbfbe456e56\" (UID: \"842bc59d-e343-4d92-9535-dcbfbe456e56\") "
Jul 28 18:18:55.523230 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:55.523198    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842bc59d-e343-4d92-9535-dcbfbe456e56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "842bc59d-e343-4d92-9535-dcbfbe456e56" (UID: "842bc59d-e343-4d92-9535-dcbfbe456e56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:18:55.523342 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:55.523242    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842bc59d-e343-4d92-9535-dcbfbe456e56-kube-api-access-cq6fl" (OuterVolumeSpecName: "kube-api-access-cq6fl") pod "842bc59d-e343-4d92-9535-dcbfbe456e56" (UID: "842bc59d-e343-4d92-9535-dcbfbe456e56"). InnerVolumeSpecName "kube-api-access-cq6fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:18:55.621759 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:55.621710    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842bc59d-e343-4d92-9535-dcbfbe456e56-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:18:55.621759 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:55.621752    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq6fl\" (UniqueName: \"kubernetes.io/projected/842bc59d-e343-4d92-9535-dcbfbe456e56-kube-api-access-cq6fl\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:18:56.162971 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:56.162928    2559 generic.go:356] "Generic (PLEG): container finished" podID="842bc59d-e343-4d92-9535-dcbfbe456e56" containerID="a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7" exitCode=0
Jul 28 18:18:56.162971 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:56.162972    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"
Jul 28 18:18:56.163522 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:56.163014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57" event={"ID":"842bc59d-e343-4d92-9535-dcbfbe456e56","Type":"ContainerDied","Data":"a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7"}
Jul 28 18:18:56.163522 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:56.163084    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57" event={"ID":"842bc59d-e343-4d92-9535-dcbfbe456e56","Type":"ContainerDied","Data":"a6497adc9eaecc070e465ed67c77daf8ae59ca10a2bb0c2db45db4d3a6230304"}
Jul 28 18:18:56.163522 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:56.163116    2559 scope.go:117] "RemoveContainer" containerID="a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7"
Jul 28 18:18:56.171418 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:56.171401    2559 scope.go:117] "RemoveContainer" containerID="a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7"
Jul 28 18:18:56.171700 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:18:56.171682    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7\": container with ID starting with a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7 not found: ID does not exist" containerID="a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7"
Jul 28 18:18:56.171753 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:56.171711    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7"} err="failed to get container status \"a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7\": rpc error: code = NotFound desc = could not find container \"a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7\": container with ID starting with a96926a49ad95ebc5e25571a0d295df9b99c95630cb7aec6f0053b9a1f6696e7 not found: ID does not exist"
Jul 28 18:18:56.180575 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:56.180546    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"]
Jul 28 18:18:56.183175 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:56.183149    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57"]
Jul 28 18:18:57.833041 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:57.833009    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="842bc59d-e343-4d92-9535-dcbfbe456e56" path="/var/lib/kubelet/pods/842bc59d-e343-4d92-9535-dcbfbe456e56/volumes"
Jul 28 18:18:59.492625 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:59.492593    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:59.586541 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:18:59.586513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:22.374973 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.374938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:22.478418 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.478388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:22.579663 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.579626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:22.842982 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.842894    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"]
Jul 28 18:19:22.843288 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.843269    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842bc59d-e343-4d92-9535-dcbfbe456e56" containerName="main"
Jul 28 18:19:22.843377 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.843291    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="842bc59d-e343-4d92-9535-dcbfbe456e56" containerName="main"
Jul 28 18:19:22.843432 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.843384    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="842bc59d-e343-4d92-9535-dcbfbe456e56" containerName="main"
Jul 28 18:19:22.846231 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.846211    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:19:22.853969 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.853949    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-v865s\""
Jul 28 18:19:22.853969 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.853963    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 28 18:19:22.854161 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.853971    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 28 18:19:22.854161 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.853955    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-epp-sa-dockercfg-66hd2\""
Jul 28 18:19:22.854161 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.853949    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 28 18:19:22.861063 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.861040    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"]
Jul 28 18:19:22.880960 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.880926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:22.952587 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.952542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvgs\" (UniqueName: \"kubernetes.io/projected/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-kube-api-access-qrvgs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq\" (UID: \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:19:22.952770 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:22.952609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq\" (UID: \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:19:23.053987 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:23.053940    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvgs\" (UniqueName: \"kubernetes.io/projected/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-kube-api-access-qrvgs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq\" (UID: \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:19:23.054198 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:23.054019    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq\" (UID: \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:19:23.056636 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:23.056612    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq\" (UID: \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:19:23.065784 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:23.065759    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvgs\" (UniqueName: \"kubernetes.io/projected/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-kube-api-access-qrvgs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq\" (UID: \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:19:23.155898 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:23.155835    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:19:23.298896 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:23.298840    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"]
Jul 28 18:19:23.301806 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:19:23.301779    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1080e2_030c_45c7_b7b4_ed1b20cdb0fe.slice/crio-0f1fb684e0f2a2c55cc4f0d92f8c5f06e13d7846bdaf2017f74c375ba7cb6c2d WatchSource:0}: Error finding container 0f1fb684e0f2a2c55cc4f0d92f8c5f06e13d7846bdaf2017f74c375ba7cb6c2d: Status 404 returned error can't find the container with id 0f1fb684e0f2a2c55cc4f0d92f8c5f06e13d7846bdaf2017f74c375ba7cb6c2d
Jul 28 18:19:24.283941 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:24.283903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq" event={"ID":"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe","Type":"ContainerStarted","Data":"bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e"}
Jul 28 18:19:24.284350 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:24.283964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq" event={"ID":"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe","Type":"ContainerStarted","Data":"0f1fb684e0f2a2c55cc4f0d92f8c5f06e13d7846bdaf2017f74c375ba7cb6c2d"}
Jul 28 18:19:24.284350 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:24.283991    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:19:24.304665 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:24.304602    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq" podStartSLOduration=2.30458129 podStartE2EDuration="2.30458129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:19:24.303260771 +0000 UTC m=+3404.957755745" watchObservedRunningTime="2026-07-28 18:19:24.30458129 +0000 UTC m=+3404.959076267"
Jul 28 18:19:35.853253 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:35.853214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:55.290610 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:19:55.290581    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:22:20.403822 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:22:20.403784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:27.839482 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:22:27.839445    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:27.939843 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:22:27.939807    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:01.976090 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:23:01.976051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:02.067700 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:23:02.067667    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:28.716314 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:23:28.716284    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f_e40a0463-3c22-4106-a788-24bb2ce11bfc/main/0.log"
Jul 28 18:23:30.945285 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:23:30.945252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:32.484753 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:23:32.484713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:33.486197 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:23:33.486169    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:33.555501 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:23:33.555454    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:34.305391 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:23:34.305355    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:13.124972 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:13.124864    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:25.065535 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.065495    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:25.162544 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.162508    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:25.251240 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.251202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:25.271332 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.271302    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 18:24:25.274400 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.274383    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.277954 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.277933    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 28 18:24:25.278253 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.278224    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-xgkv6\""
Jul 28 18:24:25.278253 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.278241    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zcd55\""
Jul 28 18:24:25.278422 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.278246    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 18:24:25.278422 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.278235    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 18:24:25.288199 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.288175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 18:24:25.321437 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.321359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.321437 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.321394    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.321618 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.321471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.321618 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.321507    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q954\" (UniqueName: \"kubernetes.io/projected/aededc8c-da83-4ecb-a232-8909e19cfa09-kube-api-access-2q954\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.321618 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.321567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.321618 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.321592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aededc8c-da83-4ecb-a232-8909e19cfa09-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.422658 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.422621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.422836 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.422665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aededc8c-da83-4ecb-a232-8909e19cfa09-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.422836 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.422686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.422836 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.422711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.422836 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.422783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.422836 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.422815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q954\" (UniqueName: \"kubernetes.io/projected/aededc8c-da83-4ecb-a232-8909e19cfa09-kube-api-access-2q954\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.423126 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.423037    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.423190 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.423160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.423245 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.423205    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.425096 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.425077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.425380 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.425362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aededc8c-da83-4ecb-a232-8909e19cfa09-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.433800 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.433776    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q954\" (UniqueName: \"kubernetes.io/projected/aededc8c-da83-4ecb-a232-8909e19cfa09-kube-api-access-2q954\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:24:25.473427 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:25.473400    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:43.729726 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:24:43.729690    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:28.284935 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:26:28.284858    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="aededc8c-da83-4ecb-a232-8909e19cfa09"
Jul 28 18:26:29.094845 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:29.094810    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:26:38.070770 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:38.070733    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:38.132263 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:38.132223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:54.786287 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:54.786249    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 18:26:54.786793 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:26:54.786503    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context canceled" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="aededc8c-da83-4ecb-a232-8909e19cfa09"
Jul 28 18:26:55.202308 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.202272    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:26:55.208649 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.208627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:26:55.333514 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.333478    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aededc8c-da83-4ecb-a232-8909e19cfa09-tls-certs\") pod \"aededc8c-da83-4ecb-a232-8909e19cfa09\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") "
Jul 28 18:26:55.333707 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.333524    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-dshm\") pod \"aededc8c-da83-4ecb-a232-8909e19cfa09\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") "
Jul 28 18:26:55.333707 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.333550    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-tmp-dir\") pod \"aededc8c-da83-4ecb-a232-8909e19cfa09\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") "
Jul 28 18:26:55.333707 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.333636    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q954\" (UniqueName: \"kubernetes.io/projected/aededc8c-da83-4ecb-a232-8909e19cfa09-kube-api-access-2q954\") pod \"aededc8c-da83-4ecb-a232-8909e19cfa09\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") "
Jul 28 18:26:55.333707 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.333662    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-model-cache\") pod \"aededc8c-da83-4ecb-a232-8909e19cfa09\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") "
Jul 28 18:26:55.333707 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.333697    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-home\") pod \"aededc8c-da83-4ecb-a232-8909e19cfa09\" (UID: \"aededc8c-da83-4ecb-a232-8909e19cfa09\") "
Jul 28 18:26:55.333985 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.333855    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aededc8c-da83-4ecb-a232-8909e19cfa09" (UID: "aededc8c-da83-4ecb-a232-8909e19cfa09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:55.334050 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.333976    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-model-cache" (OuterVolumeSpecName: "model-cache") pod "aededc8c-da83-4ecb-a232-8909e19cfa09" (UID: "aededc8c-da83-4ecb-a232-8909e19cfa09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:55.334050 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.334005    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-tmp-dir\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:26:55.334120 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.334044    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-home" (OuterVolumeSpecName: "home") pod "aededc8c-da83-4ecb-a232-8909e19cfa09" (UID: "aededc8c-da83-4ecb-a232-8909e19cfa09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:55.335795 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.335774    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-dshm" (OuterVolumeSpecName: "dshm") pod "aededc8c-da83-4ecb-a232-8909e19cfa09" (UID: "aededc8c-da83-4ecb-a232-8909e19cfa09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:55.335863 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.335797    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aededc8c-da83-4ecb-a232-8909e19cfa09-kube-api-access-2q954" (OuterVolumeSpecName: "kube-api-access-2q954") pod "aededc8c-da83-4ecb-a232-8909e19cfa09" (UID: "aededc8c-da83-4ecb-a232-8909e19cfa09"). InnerVolumeSpecName "kube-api-access-2q954". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:26:55.335863 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.335847    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aededc8c-da83-4ecb-a232-8909e19cfa09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aededc8c-da83-4ecb-a232-8909e19cfa09" (UID: "aededc8c-da83-4ecb-a232-8909e19cfa09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:26:55.435245 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.435213    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aededc8c-da83-4ecb-a232-8909e19cfa09-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:26:55.435245 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.435238    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-dshm\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:26:55.435245 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.435247    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q954\" (UniqueName: \"kubernetes.io/projected/aededc8c-da83-4ecb-a232-8909e19cfa09-kube-api-access-2q954\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:26:55.435475 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.435256    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-model-cache\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:26:55.435475 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:55.435264    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aededc8c-da83-4ecb-a232-8909e19cfa09-home\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:26:56.206106 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:56.206077    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 18:26:56.236239 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:56.236206    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 18:26:56.239477 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:56.239452    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 18:26:57.833216 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:57.833170    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aededc8c-da83-4ecb-a232-8909e19cfa09" path="/var/lib/kubelet/pods/aededc8c-da83-4ecb-a232-8909e19cfa09/volumes"
Jul 28 18:26:58.054168 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:58.054132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:58.153560 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:58.153528    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:58.251992 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:26:58.251956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:00.659550 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:00.659515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:00.760385 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:00.760345    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:03.240133 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:03.240101    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:03.343701 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:03.343664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:04.683300 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:04.683258    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:06.641851 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:06.641816    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:06.741986 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:06.741952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:07.323696 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:07.323658    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:07.425992 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:07.425959    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:07.528792 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:07.528758    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:10.483120 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:10.483083    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:10.547702 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:10.547667    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:10.714261 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:10.714218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:10.776364 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:10.776265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:11.641054 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:11.641025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:11.742749 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:11.742711    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:11.842170 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:11.842138    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:13.487282 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:13.487253    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:13.542121 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:13.542084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:16.143800 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:16.143767    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:16.373708 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:16.373663    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:16.474036 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:16.473938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:18.362832 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:18.362804    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:18.471958 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:18.471923    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:21.613284 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:21.613244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:21.713575 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:21.713541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:21.814987 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:21.814928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:22.728929 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:22.728856    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:22.847531 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:22.847483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:22.925186 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:22.925152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:23.026674 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:23.026584    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:23.126987 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:23.126955    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:23.648097 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:23.648068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:23.746575 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:23.746543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:23.853312 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:23.853280    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:24.820023 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:24.819988    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:24.917246 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:24.917217    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:25.017013 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:25.016981    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:26.001025 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:26.000995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:26.101806 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:26.101771    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:26.200131 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:26.200102    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:27.146137 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:27.146103    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:27.235919 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:27.235887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:28.340694 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:28.340661    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:28.441819 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:28.441767    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:28.963265 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:28.963230    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:30.215107 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:30.215070    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:30.315598 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:30.315557    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:31.319258 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:31.319222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:31.419263 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:31.419231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:32.214508 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:32.214471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:32.315869 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:32.315838    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:33.618806 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:33.618771    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:33.685268 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:33.685239    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:46.707299 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.707261    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:46.934650 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.934610    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"]
Jul 28 18:27:46.937858 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.937837    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:27:46.940930 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.940894    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 18:27:46.941072 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.940932    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 18:27:46.941072 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.940944    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-djhgl\""
Jul 28 18:27:46.941072 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.940932    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 18:27:46.941072 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.940981    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-4gvhr\""
Jul 28 18:27:46.947525 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.947506    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"]
Jul 28 18:27:46.955343 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:46.955318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:47.067278 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:47.067197    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9b6\" (UniqueName: \"kubernetes.io/projected/05997b19-9aa3-410c-8ae7-144ae4add814-kube-api-access-qk9b6\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69\" (UID: \"05997b19-9aa3-410c-8ae7-144ae4add814\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:27:47.067422 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:47.067295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05997b19-9aa3-410c-8ae7-144ae4add814-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69\" (UID: \"05997b19-9aa3-410c-8ae7-144ae4add814\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:27:47.168536 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:47.168499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05997b19-9aa3-410c-8ae7-144ae4add814-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69\" (UID: \"05997b19-9aa3-410c-8ae7-144ae4add814\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:27:47.168711 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:47.168570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9b6\" (UniqueName: \"kubernetes.io/projected/05997b19-9aa3-410c-8ae7-144ae4add814-kube-api-access-qk9b6\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69\" (UID: \"05997b19-9aa3-410c-8ae7-144ae4add814\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:27:47.171221 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:47.171200    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05997b19-9aa3-410c-8ae7-144ae4add814-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69\" (UID: \"05997b19-9aa3-410c-8ae7-144ae4add814\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:27:47.183401 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:47.183374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9b6\" (UniqueName: \"kubernetes.io/projected/05997b19-9aa3-410c-8ae7-144ae4add814-kube-api-access-qk9b6\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69\" (UID: \"05997b19-9aa3-410c-8ae7-144ae4add814\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:27:47.248217 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:47.248178    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:27:47.387406 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:47.387374    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"]
Jul 28 18:27:47.390324 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:27:47.390294    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05997b19_9aa3_410c_8ae7_144ae4add814.slice/crio-3717f70fd88ef094c1cb24e34be7acdc8245655a19c9bb5a49c41309ee8c8206 WatchSource:0}: Error finding container 3717f70fd88ef094c1cb24e34be7acdc8245655a19c9bb5a49c41309ee8c8206: Status 404 returned error can't find the container with id 3717f70fd88ef094c1cb24e34be7acdc8245655a19c9bb5a49c41309ee8c8206
Jul 28 18:27:47.424178 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:47.424149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69" event={"ID":"05997b19-9aa3-410c-8ae7-144ae4add814","Type":"ContainerStarted","Data":"3717f70fd88ef094c1cb24e34be7acdc8245655a19c9bb5a49c41309ee8c8206"}
Jul 28 18:27:48.430971 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:48.430933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69" event={"ID":"05997b19-9aa3-410c-8ae7-144ae4add814","Type":"ContainerStarted","Data":"e9fb74578225338e4c98225edd0528f250a41285df75957801ca21b42837bc6d"}
Jul 28 18:27:48.431374 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:48.431062    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:27:48.450210 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:48.450152    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69" podStartSLOduration=2.450131478 podStartE2EDuration="2.450131478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:27:48.447734655 +0000 UTC m=+3909.102229631" watchObservedRunningTime="2026-07-28 18:27:48.450131478 +0000 UTC m=+3909.104626454"
Jul 28 18:27:59.269810 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:59.269777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:59.330284 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:27:59.330244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:19.417982 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:28:19.417944    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:19.439031 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:28:19.439001    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-7lw69"
Jul 28 18:29:02.239077 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:29:02.239045    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:03.721106 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:29:03.721063    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:05.270793 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:29:05.270751    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:05.341430 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:29:05.341389    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:13.234245 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:13.234206    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"]
Jul 28 18:34:13.254241 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:13.254205    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:14.718949 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:14.718903    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:15.290566 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:15.290516    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq" podUID="7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:34:25.289780 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:25.289738    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq" podUID="7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:34:28.254443 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:28.254394    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq" podUID="7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" containerName="main" containerID="cri-o://bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e" gracePeriod=45
Jul 28 18:34:28.496490 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:28.496465    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:34:28.519202 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:28.519117    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrvgs\" (UniqueName: \"kubernetes.io/projected/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-kube-api-access-qrvgs\") pod \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\" (UID: \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\") "
Jul 28 18:34:28.519202 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:28.519176    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-tls-certs\") pod \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\" (UID: \"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe\") "
Jul 28 18:34:28.521438 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:28.521403    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" (UID: "7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:34:28.521588 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:28.521462    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-kube-api-access-qrvgs" (OuterVolumeSpecName: "kube-api-access-qrvgs") pod "7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" (UID: "7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe"). InnerVolumeSpecName "kube-api-access-qrvgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:28.620170 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:28.620130    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrvgs\" (UniqueName: \"kubernetes.io/projected/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-kube-api-access-qrvgs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:34:28.620170 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:28.620163    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:34:29.144339 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.144304    2559 generic.go:356] "Generic (PLEG): container finished" podID="7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" containerID="bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e" exitCode=0
Jul 28 18:34:29.144526 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.144352    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"
Jul 28 18:34:29.144526 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.144389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq" event={"ID":"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe","Type":"ContainerDied","Data":"bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e"}
Jul 28 18:34:29.144526 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.144450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq" event={"ID":"7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe","Type":"ContainerDied","Data":"0f1fb684e0f2a2c55cc4f0d92f8c5f06e13d7846bdaf2017f74c375ba7cb6c2d"}
Jul 28 18:34:29.144526 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.144481    2559 scope.go:117] "RemoveContainer" containerID="bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e"
Jul 28 18:34:29.153334 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.153316    2559 scope.go:117] "RemoveContainer" containerID="bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e"
Jul 28 18:34:29.153607 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:34:29.153577    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e\": container with ID starting with bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e not found: ID does not exist" containerID="bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e"
Jul 28 18:34:29.153675 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.153608    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e"} err="failed to get container status \"bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e\": rpc error: code = NotFound desc = could not find container \"bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e\": container with ID starting with bfb44bddc2c4cd7e3ec6b9efca0fd1aa5806d2693a99296d56604b355a21111e not found: ID does not exist"
Jul 28 18:34:29.165199 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.165169    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"]
Jul 28 18:34:29.169650 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.169626    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq"]
Jul 28 18:34:29.833142 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:29.833107    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" path="/var/lib/kubelet/pods/7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe/volumes"
Jul 28 18:34:32.234712 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:32.234668    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:32.336064 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:32.336021    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:46.835123 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:46.835089    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:46.941306 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:46.941272    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:47.023179 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.023144    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:47.139432 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.139400    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"]
Jul 28 18:34:47.139737 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.139724    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" containerName="main"
Jul 28 18:34:47.139791 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.139740    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" containerName="main"
Jul 28 18:34:47.139827 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.139795    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1080e2-030c-45c7-b7b4-ed1b20cdb0fe" containerName="main"
Jul 28 18:34:47.143687 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.143667    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:34:47.146037 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.146014    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 28 18:34:47.146726 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.146692    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 28 18:34:47.146726 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.146712    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-qr8kw\""
Jul 28 18:34:47.146939 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.146742    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 28 18:34:47.146939 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.146787    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-epp-sa-dockercfg-k9sgv\""
Jul 28 18:34:47.155230 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.155208    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"]
Jul 28 18:34:47.176999 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.176975    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:47.179045 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.179024    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae99fd5-f76f-449e-9f89-d06d6f444200-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w\" (UID: \"1ae99fd5-f76f-449e-9f89-d06d6f444200\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:34:47.179150 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.179059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskkd\" (UniqueName: \"kubernetes.io/projected/1ae99fd5-f76f-449e-9f89-d06d6f444200-kube-api-access-tskkd\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w\" (UID: \"1ae99fd5-f76f-449e-9f89-d06d6f444200\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:34:47.280238 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.280206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae99fd5-f76f-449e-9f89-d06d6f444200-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w\" (UID: \"1ae99fd5-f76f-449e-9f89-d06d6f444200\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:34:47.280402 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.280244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskkd\" (UniqueName: \"kubernetes.io/projected/1ae99fd5-f76f-449e-9f89-d06d6f444200-kube-api-access-tskkd\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w\" (UID: \"1ae99fd5-f76f-449e-9f89-d06d6f444200\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:34:47.282915 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.282892    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae99fd5-f76f-449e-9f89-d06d6f444200-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w\" (UID: \"1ae99fd5-f76f-449e-9f89-d06d6f444200\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:34:47.287780 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.287753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskkd\" (UniqueName: \"kubernetes.io/projected/1ae99fd5-f76f-449e-9f89-d06d6f444200-kube-api-access-tskkd\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w\" (UID: \"1ae99fd5-f76f-449e-9f89-d06d6f444200\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:34:47.454966 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.454855    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:34:47.588304 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:47.588278    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"]
Jul 28 18:34:47.590697 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:34:47.590671    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae99fd5_f76f_449e_9f89_d06d6f444200.slice/crio-d6a7eed8292ad228c3f3acd0871d902ec3712ac01bfaf0b93781e388ec37e587 WatchSource:0}: Error finding container d6a7eed8292ad228c3f3acd0871d902ec3712ac01bfaf0b93781e388ec37e587: Status 404 returned error can't find the container with id d6a7eed8292ad228c3f3acd0871d902ec3712ac01bfaf0b93781e388ec37e587
Jul 28 18:34:48.227549 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:48.227505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w" event={"ID":"1ae99fd5-f76f-449e-9f89-d06d6f444200","Type":"ContainerStarted","Data":"7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015"}
Jul 28 18:34:48.228013 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:48.227576    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w" event={"ID":"1ae99fd5-f76f-449e-9f89-d06d6f444200","Type":"ContainerStarted","Data":"d6a7eed8292ad228c3f3acd0871d902ec3712ac01bfaf0b93781e388ec37e587"}
Jul 28 18:34:48.228013 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:48.227716    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:34:48.246902 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:34:48.246818    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w" podStartSLOduration=1.246799059 podStartE2EDuration="1.246799059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:48.245493226 +0000 UTC m=+4328.899988203" watchObservedRunningTime="2026-07-28 18:34:48.246799059 +0000 UTC m=+4328.901294036"
Jul 28 18:35:03.150332 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:35:03.150297    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:19.234033 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:35:19.233999    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:35:33.555464 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:35:33.555424    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:33.656717 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:35:33.656678    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:36.314694 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:36:36.314657    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:07.521039 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:38:07.520992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:07.575517 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:38:07.575441    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:04.745240 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:39:04.745155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:42:29.341469 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:42:29.341370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:42:30.904417 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:42:30.904384    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:43:09.706092 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:43:09.706038    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:06.147950 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:06.147846    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:14.452652 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:14.452582    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:14.552411 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:14.552381    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:25.132316 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:25.132274    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:25.233091 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:25.233059    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:25.305989 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:25.305952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:31.768320 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:31.768273    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:31.868829 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:31.868799    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:32.365309 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:32.365279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:41.854369 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:45:41.854328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:00.687901 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:48:00.687850    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:22.614581 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:48:22.614550    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:41.832917 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:48:41.832869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:41.934078 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:48:41.934039    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:42.033324 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:48:42.033291    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:47.993527 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:48:47.993491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:48:48.057892 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:48:48.057843    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:07.959318 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:07.959282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:08.149545 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.149506    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"]
Jul 28 18:49:08.152213 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.152190    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:08.157370 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.157339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 28 18:49:08.157537 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.157429    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-mdddw\""
Jul 28 18:49:08.157537 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.157457    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-qs25j\""
Jul 28 18:49:08.157537 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.157457    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 28 18:49:08.157713 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.157558    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 28 18:49:08.164479 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.164455    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"]
Jul 28 18:49:08.193029 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.192994    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:08.279487 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.279392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e828da3-9a84-49a5-b6b6-da44215a0cc0-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-r266d\" (UID: \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:08.279487 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.279436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvnv\" (UniqueName: \"kubernetes.io/projected/6e828da3-9a84-49a5-b6b6-da44215a0cc0-kube-api-access-6bvnv\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-r266d\" (UID: \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:08.380381 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.380343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e828da3-9a84-49a5-b6b6-da44215a0cc0-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-r266d\" (UID: \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:08.380381 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.380380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvnv\" (UniqueName: \"kubernetes.io/projected/6e828da3-9a84-49a5-b6b6-da44215a0cc0-kube-api-access-6bvnv\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-r266d\" (UID: \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:08.383053 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.383026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e828da3-9a84-49a5-b6b6-da44215a0cc0-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-r266d\" (UID: \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:08.388229 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.388205    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvnv\" (UniqueName: \"kubernetes.io/projected/6e828da3-9a84-49a5-b6b6-da44215a0cc0-kube-api-access-6bvnv\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-r266d\" (UID: \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:08.465129 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.465087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:08.601593 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:08.601565    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"]
Jul 28 18:49:08.604652 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:49:08.604622    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e828da3_9a84_49a5_b6b6_da44215a0cc0.slice/crio-9b8edd558f287734eb1d8867f1bfb148366adc923d4dc7934954fdc9a73e675e WatchSource:0}: Error finding container 9b8edd558f287734eb1d8867f1bfb148366adc923d4dc7934954fdc9a73e675e: Status 404 returned error can't find the container with id 9b8edd558f287734eb1d8867f1bfb148366adc923d4dc7934954fdc9a73e675e
Jul 28 18:49:09.010244 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:09.010204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d" event={"ID":"6e828da3-9a84-49a5-b6b6-da44215a0cc0","Type":"ContainerStarted","Data":"454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353"}
Jul 28 18:49:09.010685 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:09.010255    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d" event={"ID":"6e828da3-9a84-49a5-b6b6-da44215a0cc0","Type":"ContainerStarted","Data":"9b8edd558f287734eb1d8867f1bfb148366adc923d4dc7934954fdc9a73e675e"}
Jul 28 18:49:09.010685 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:09.010287    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:09.027642 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:09.027591    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d" podStartSLOduration=1.027571658 podStartE2EDuration="1.027571658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:49:09.026788212 +0000 UTC m=+5189.681283187" watchObservedRunningTime="2026-07-28 18:49:09.027571658 +0000 UTC m=+5189.682066632"
Jul 28 18:49:30.761952 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:30.761914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:40.016158 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:40.016123    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:49:41.654511 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:41.654475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:41.676802 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:41.676768    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"]
Jul 28 18:49:41.719496 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:41.719465    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:49.234582 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:49.234539    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w" podUID="1ae99fd5-f76f-449e-9f89-d06d6f444200" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:49:56.696323 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:56.696281    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w" podUID="1ae99fd5-f76f-449e-9f89-d06d6f444200" containerName="main" containerID="cri-o://7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015" gracePeriod=45
Jul 28 18:49:56.949454 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:56.949384    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:49:57.099983 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.099946    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae99fd5-f76f-449e-9f89-d06d6f444200-tls-certs\") pod \"1ae99fd5-f76f-449e-9f89-d06d6f444200\" (UID: \"1ae99fd5-f76f-449e-9f89-d06d6f444200\") "
Jul 28 18:49:57.100156 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.099994    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tskkd\" (UniqueName: \"kubernetes.io/projected/1ae99fd5-f76f-449e-9f89-d06d6f444200-kube-api-access-tskkd\") pod \"1ae99fd5-f76f-449e-9f89-d06d6f444200\" (UID: \"1ae99fd5-f76f-449e-9f89-d06d6f444200\") "
Jul 28 18:49:57.102387 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.102348    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae99fd5-f76f-449e-9f89-d06d6f444200-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ae99fd5-f76f-449e-9f89-d06d6f444200" (UID: "1ae99fd5-f76f-449e-9f89-d06d6f444200"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:49:57.102387 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.102354    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae99fd5-f76f-449e-9f89-d06d6f444200-kube-api-access-tskkd" (OuterVolumeSpecName: "kube-api-access-tskkd") pod "1ae99fd5-f76f-449e-9f89-d06d6f444200" (UID: "1ae99fd5-f76f-449e-9f89-d06d6f444200"). InnerVolumeSpecName "kube-api-access-tskkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:49:57.201450 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.201373    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tskkd\" (UniqueName: \"kubernetes.io/projected/1ae99fd5-f76f-449e-9f89-d06d6f444200-kube-api-access-tskkd\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:49:57.201450 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.201400    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae99fd5-f76f-449e-9f89-d06d6f444200-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:49:57.230427 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.230388    2559 generic.go:356] "Generic (PLEG): container finished" podID="1ae99fd5-f76f-449e-9f89-d06d6f444200" containerID="7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015" exitCode=0
Jul 28 18:49:57.230552 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.230437    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"
Jul 28 18:49:57.230552 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.230471    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w" event={"ID":"1ae99fd5-f76f-449e-9f89-d06d6f444200","Type":"ContainerDied","Data":"7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015"}
Jul 28 18:49:57.230552 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.230546    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w" event={"ID":"1ae99fd5-f76f-449e-9f89-d06d6f444200","Type":"ContainerDied","Data":"d6a7eed8292ad228c3f3acd0871d902ec3712ac01bfaf0b93781e388ec37e587"}
Jul 28 18:49:57.230653 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.230568    2559 scope.go:117] "RemoveContainer" containerID="7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015"
Jul 28 18:49:57.239313 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.239292    2559 scope.go:117] "RemoveContainer" containerID="7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015"
Jul 28 18:49:57.239518 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:49:57.239501    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015\": container with ID starting with 7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015 not found: ID does not exist" containerID="7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015"
Jul 28 18:49:57.239570 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.239526    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015"} err="failed to get container status \"7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015\": rpc error: code = NotFound desc = could not find container \"7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015\": container with ID starting with 7f5d1a55d05a808713db2eb47a6b29ac1325ae9d532fa32bad61937e15954015 not found: ID does not exist"
Jul 28 18:49:57.251428 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.251405    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"]
Jul 28 18:49:57.257337 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.257314    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w"]
Jul 28 18:49:57.832643 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:49:57.832595    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae99fd5-f76f-449e-9f89-d06d6f444200" path="/var/lib/kubelet/pods/1ae99fd5-f76f-449e-9f89-d06d6f444200/volumes"
Jul 28 18:50:00.203632 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:00.203597    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:50:00.303433 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:00.303398    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:50:48.331552 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.331516    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:50:48.536456 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.536420    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"]
Jul 28 18:50:48.536771 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.536756    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae99fd5-f76f-449e-9f89-d06d6f444200" containerName="main"
Jul 28 18:50:48.536771 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.536771    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae99fd5-f76f-449e-9f89-d06d6f444200" containerName="main"
Jul 28 18:50:48.536944 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.536822    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae99fd5-f76f-449e-9f89-d06d6f444200" containerName="main"
Jul 28 18:50:48.541150 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.541128    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:50:48.544235 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.544207    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"autoscale-cleanup-hpa-kserve-self-signed-certs\""
Jul 28 18:50:48.544369 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.544256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"autoscale-cleanup-hpa-epp-sa-dockercfg-2nfjr\""
Jul 28 18:50:48.544369 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.544262    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"kube-root-ca.crt\""
Jul 28 18:50:48.544369 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.544256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"default-dockercfg-frqjv\""
Jul 28 18:50:48.544604 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.544577    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"openshift-service-ca.crt\""
Jul 28 18:50:48.551084 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.551062    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"]
Jul 28 18:50:48.575694 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.575656    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:50:48.619912 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.619859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5\" (UID: \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:50:48.620093 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.619933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8sbw\" (UniqueName: \"kubernetes.io/projected/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-kube-api-access-m8sbw\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5\" (UID: \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:50:48.721100 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.721062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5\" (UID: \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:50:48.721283 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.721118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8sbw\" (UniqueName: \"kubernetes.io/projected/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-kube-api-access-m8sbw\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5\" (UID: \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:50:48.723768 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.723738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5\" (UID: \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:50:48.730503 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.730482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8sbw\" (UniqueName: \"kubernetes.io/projected/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-kube-api-access-m8sbw\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5\" (UID: \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:50:48.852340 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.852301    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:50:48.990997 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:48.990962    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"]
Jul 28 18:50:48.994248 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:50:48.994223    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c19ee5c_5ca7_48bd_9a1f_951af3fbbf47.slice/crio-d37fc85713eaafbb28608557d559f5b402226abd58eeadc03922d3506baa921f WatchSource:0}: Error finding container d37fc85713eaafbb28608557d559f5b402226abd58eeadc03922d3506baa921f: Status 404 returned error can't find the container with id d37fc85713eaafbb28608557d559f5b402226abd58eeadc03922d3506baa921f
Jul 28 18:50:49.460783 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:49.460750    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5" event={"ID":"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47","Type":"ContainerStarted","Data":"b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d"}
Jul 28 18:50:49.461308 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:49.460803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5" event={"ID":"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47","Type":"ContainerStarted","Data":"d37fc85713eaafbb28608557d559f5b402226abd58eeadc03922d3506baa921f"}
Jul 28 18:50:49.461308 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:49.460868    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:50:49.480767 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:50:49.480716    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5" podStartSLOduration=1.4807018730000001 podStartE2EDuration="1.480701873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:50:49.479179486 +0000 UTC m=+5290.133674463" watchObservedRunningTime="2026-07-28 18:50:49.480701873 +0000 UTC m=+5290.135196872"
Jul 28 18:51:08.996738 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:51:08.996655    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:51:11.829499 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:51:11.829464    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:51:11.887581 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:51:11.887542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:51:20.467442 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:51:20.467404    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 18:52:07.174112 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:07.174071    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:07.232460 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:07.232428    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:13.055472 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:13.055433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:13.061490 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:13.061461    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"]
Jul 28 18:52:13.116028 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:13.115985    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:20.016080 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:20.016042    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d" podUID="6e828da3-9a84-49a5-b6b6-da44215a0cc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:52:28.081115 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.081069    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d" podUID="6e828da3-9a84-49a5-b6b6-da44215a0cc0" containerName="main" containerID="cri-o://454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353" gracePeriod=45
Jul 28 18:52:28.333381 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.333321    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:52:28.449443 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.449411    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e828da3-9a84-49a5-b6b6-da44215a0cc0-tls-certs\") pod \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\" (UID: \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\") "
Jul 28 18:52:28.449443 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.449449    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bvnv\" (UniqueName: \"kubernetes.io/projected/6e828da3-9a84-49a5-b6b6-da44215a0cc0-kube-api-access-6bvnv\") pod \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\" (UID: \"6e828da3-9a84-49a5-b6b6-da44215a0cc0\") "
Jul 28 18:52:28.451861 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.451823    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e828da3-9a84-49a5-b6b6-da44215a0cc0-kube-api-access-6bvnv" (OuterVolumeSpecName: "kube-api-access-6bvnv") pod "6e828da3-9a84-49a5-b6b6-da44215a0cc0" (UID: "6e828da3-9a84-49a5-b6b6-da44215a0cc0"). InnerVolumeSpecName "kube-api-access-6bvnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:52:28.452031 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.451908    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e828da3-9a84-49a5-b6b6-da44215a0cc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e828da3-9a84-49a5-b6b6-da44215a0cc0" (UID: "6e828da3-9a84-49a5-b6b6-da44215a0cc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:52:28.550410 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.550375    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e828da3-9a84-49a5-b6b6-da44215a0cc0-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:52:28.550410 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.550403    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvnv\" (UniqueName: \"kubernetes.io/projected/6e828da3-9a84-49a5-b6b6-da44215a0cc0-kube-api-access-6bvnv\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:52:28.900259 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.900218    2559 generic.go:356] "Generic (PLEG): container finished" podID="6e828da3-9a84-49a5-b6b6-da44215a0cc0" containerID="454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353" exitCode=0
Jul 28 18:52:28.900474 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.900273    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d" event={"ID":"6e828da3-9a84-49a5-b6b6-da44215a0cc0","Type":"ContainerDied","Data":"454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353"}
Jul 28 18:52:28.900474 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.900281    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"
Jul 28 18:52:28.900474 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.900338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d" event={"ID":"6e828da3-9a84-49a5-b6b6-da44215a0cc0","Type":"ContainerDied","Data":"9b8edd558f287734eb1d8867f1bfb148366adc923d4dc7934954fdc9a73e675e"}
Jul 28 18:52:28.900474 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.900359    2559 scope.go:117] "RemoveContainer" containerID="454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353"
Jul 28 18:52:28.910068 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.910042    2559 scope.go:117] "RemoveContainer" containerID="454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353"
Jul 28 18:52:28.910356 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:52:28.910332    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353\": container with ID starting with 454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353 not found: ID does not exist" containerID="454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353"
Jul 28 18:52:28.910445 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.910363    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353"} err="failed to get container status \"454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353\": rpc error: code = NotFound desc = could not find container \"454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353\": container with ID starting with 454cad8b5662f57c57ccf253690689d601a4e7dfce8de607435b0e7e24aba353 not found: ID does not exist"
Jul 28 18:52:28.920833 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.920805    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"]
Jul 28 18:52:28.927077 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:28.927051    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-r266d"]
Jul 28 18:52:29.771963 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:29.771928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:29.833481 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:29.833448    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e828da3-9a84-49a5-b6b6-da44215a0cc0" path="/var/lib/kubelet/pods/6e828da3-9a84-49a5-b6b6-da44215a0cc0/volumes"
Jul 28 18:52:29.871849 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:29.871815    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:29.936814 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:29.936773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:35.676082 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:35.676038    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:36.140481 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:36.140451    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:41.921754 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:41.921712    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:41.981028 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:41.980998    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:51.719515 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.719483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:51.962283 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.962255    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"]
Jul 28 18:52:51.962693 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.962662    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e828da3-9a84-49a5-b6b6-da44215a0cc0" containerName="main"
Jul 28 18:52:51.962693 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.962686    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e828da3-9a84-49a5-b6b6-da44215a0cc0" containerName="main"
Jul 28 18:52:51.962909 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.962786    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e828da3-9a84-49a5-b6b6-da44215a0cc0" containerName="main"
Jul 28 18:52:51.966839 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.966820    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:52:51.969998 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.969923    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 18:52:51.969998 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.969944    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-bv5vb\""
Jul 28 18:52:51.970160 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.969931    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4nwj4\""
Jul 28 18:52:51.970160 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.969931    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 18:52:51.970160 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.969927    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 18:52:51.977273 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.977249    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"]
Jul 28 18:52:51.994919 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:51.994888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:52.040775 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:52.040744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ced98d-28c4-4186-8ea7-a25e50e66d06-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7\" (UID: \"09ced98d-28c4-4186-8ea7-a25e50e66d06\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:52:52.040962 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:52.040850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhr8\" (UniqueName: \"kubernetes.io/projected/09ced98d-28c4-4186-8ea7-a25e50e66d06-kube-api-access-qvhr8\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7\" (UID: \"09ced98d-28c4-4186-8ea7-a25e50e66d06\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:52:52.141265 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:52.141231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhr8\" (UniqueName: \"kubernetes.io/projected/09ced98d-28c4-4186-8ea7-a25e50e66d06-kube-api-access-qvhr8\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7\" (UID: \"09ced98d-28c4-4186-8ea7-a25e50e66d06\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:52:52.141436 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:52.141283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ced98d-28c4-4186-8ea7-a25e50e66d06-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7\" (UID: \"09ced98d-28c4-4186-8ea7-a25e50e66d06\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:52:52.144056 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:52.144017    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ced98d-28c4-4186-8ea7-a25e50e66d06-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7\" (UID: \"09ced98d-28c4-4186-8ea7-a25e50e66d06\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:52:52.150520 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:52.150491    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhr8\" (UniqueName: \"kubernetes.io/projected/09ced98d-28c4-4186-8ea7-a25e50e66d06-kube-api-access-qvhr8\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7\" (UID: \"09ced98d-28c4-4186-8ea7-a25e50e66d06\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:52:52.278215 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:52.278133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:52:52.412344 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:52.412302    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"]
Jul 28 18:52:52.416512 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:52:52.416484    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ced98d_28c4_4186_8ea7_a25e50e66d06.slice/crio-5e2566c38a81d350fd568274838b78b771d935e657a48b98c9ed55f61474f8ea WatchSource:0}: Error finding container 5e2566c38a81d350fd568274838b78b771d935e657a48b98c9ed55f61474f8ea: Status 404 returned error can't find the container with id 5e2566c38a81d350fd568274838b78b771d935e657a48b98c9ed55f61474f8ea
Jul 28 18:52:53.007722 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:53.007676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7" event={"ID":"09ced98d-28c4-4186-8ea7-a25e50e66d06","Type":"ContainerStarted","Data":"7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf"}
Jul 28 18:52:53.008159 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:53.007772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7" event={"ID":"09ced98d-28c4-4186-8ea7-a25e50e66d06","Type":"ContainerStarted","Data":"5e2566c38a81d350fd568274838b78b771d935e657a48b98c9ed55f61474f8ea"}
Jul 28 18:52:53.008159 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:53.007800    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:52:53.028597 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:52:53.028548    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7" podStartSLOduration=2.028532398 podStartE2EDuration="2.028532398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:52:53.026642422 +0000 UTC m=+5413.681137396" watchObservedRunningTime="2026-07-28 18:52:53.028532398 +0000 UTC m=+5413.683027402"
Jul 28 18:53:11.279948 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:53:11.279912    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:24.013806 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:53:24.013765    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:54:10.417466 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:54:10.417384    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:54:43.668712 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:54:43.668654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:55:28.044603 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:28.044568    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"]
Jul 28 18:55:28.078312 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:28.078285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:55:28.138247 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:28.138215    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:55:34.014100 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:34.014060    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7" podUID="09ced98d-28c4-4186-8ea7-a25e50e66d06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:55:43.064319 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.064282    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7" podUID="09ced98d-28c4-4186-8ea7-a25e50e66d06" containerName="main" containerID="cri-o://7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf" gracePeriod=45
Jul 28 18:55:43.319985 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.319904    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:55:43.412767 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.412726    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ced98d-28c4-4186-8ea7-a25e50e66d06-tls-certs\") pod \"09ced98d-28c4-4186-8ea7-a25e50e66d06\" (UID: \"09ced98d-28c4-4186-8ea7-a25e50e66d06\") "
Jul 28 18:55:43.412767 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.412778    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvhr8\" (UniqueName: \"kubernetes.io/projected/09ced98d-28c4-4186-8ea7-a25e50e66d06-kube-api-access-qvhr8\") pod \"09ced98d-28c4-4186-8ea7-a25e50e66d06\" (UID: \"09ced98d-28c4-4186-8ea7-a25e50e66d06\") "
Jul 28 18:55:43.415129 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.415096    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ced98d-28c4-4186-8ea7-a25e50e66d06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09ced98d-28c4-4186-8ea7-a25e50e66d06" (UID: "09ced98d-28c4-4186-8ea7-a25e50e66d06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:55:43.415129 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.415111    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ced98d-28c4-4186-8ea7-a25e50e66d06-kube-api-access-qvhr8" (OuterVolumeSpecName: "kube-api-access-qvhr8") pod "09ced98d-28c4-4186-8ea7-a25e50e66d06" (UID: "09ced98d-28c4-4186-8ea7-a25e50e66d06"). InnerVolumeSpecName "kube-api-access-qvhr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:55:43.513700 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.513665    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ced98d-28c4-4186-8ea7-a25e50e66d06-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:55:43.513700 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.513697    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvhr8\" (UniqueName: \"kubernetes.io/projected/09ced98d-28c4-4186-8ea7-a25e50e66d06-kube-api-access-qvhr8\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:55:43.734870 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.734831    2559 generic.go:356] "Generic (PLEG): container finished" podID="09ced98d-28c4-4186-8ea7-a25e50e66d06" containerID="7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf" exitCode=0
Jul 28 18:55:43.735076 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.734902    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"
Jul 28 18:55:43.735076 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.734915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7" event={"ID":"09ced98d-28c4-4186-8ea7-a25e50e66d06","Type":"ContainerDied","Data":"7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf"}
Jul 28 18:55:43.735076 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.734972    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7" event={"ID":"09ced98d-28c4-4186-8ea7-a25e50e66d06","Type":"ContainerDied","Data":"5e2566c38a81d350fd568274838b78b771d935e657a48b98c9ed55f61474f8ea"}
Jul 28 18:55:43.735076 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.734994    2559 scope.go:117] "RemoveContainer" containerID="7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf"
Jul 28 18:55:43.744124 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.744101    2559 scope.go:117] "RemoveContainer" containerID="7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf"
Jul 28 18:55:43.744425 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:55:43.744404    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf\": container with ID starting with 7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf not found: ID does not exist" containerID="7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf"
Jul 28 18:55:43.744520 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.744430    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf"} err="failed to get container status \"7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf\": rpc error: code = NotFound desc = could not find container \"7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf\": container with ID starting with 7f4fdc902be17a0b36f777e87f0326d04a0f6a5825eb79dbfcf92100b5b6e6cf not found: ID does not exist"
Jul 28 18:55:43.759796 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.759763    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"]
Jul 28 18:55:43.767237 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.767209    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-hx6h7"]
Jul 28 18:55:43.832672 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:55:43.832637    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ced98d-28c4-4186-8ea7-a25e50e66d06" path="/var/lib/kubelet/pods/09ced98d-28c4-4186-8ea7-a25e50e66d06/volumes"
Jul 28 18:56:33.058287 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:33.058251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:33.178675 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:33.178636    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:33.285488 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:33.285448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:39.215002 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.214967    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:39.566947 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.566832    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"]
Jul 28 18:56:39.567250 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.567232    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ced98d-28c4-4186-8ea7-a25e50e66d06" containerName="main"
Jul 28 18:56:39.567338 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.567254    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ced98d-28c4-4186-8ea7-a25e50e66d06" containerName="main"
Jul 28 18:56:39.567393 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.567346    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ced98d-28c4-4186-8ea7-a25e50e66d06" containerName="main"
Jul 28 18:56:39.575993 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.575965    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:56:39.582145 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.582117    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 28 18:56:39.582839 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.582821    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 28 18:56:39.582936 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.582867    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-hsm69\""
Jul 28 18:56:39.584373 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.584355    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-6fjp8\""
Jul 28 18:56:39.584507 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.584491    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 28 18:56:39.603058 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.603032    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"]
Jul 28 18:56:39.619030 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.619003    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:39.687012 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.686982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e74d2c55-eaca-48c7-aa38-236e04cf0dff-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25\" (UID: \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:56:39.687170 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.687043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfp6w\" (UniqueName: \"kubernetes.io/projected/e74d2c55-eaca-48c7-aa38-236e04cf0dff-kube-api-access-xfp6w\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25\" (UID: \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:56:39.787834 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.787803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e74d2c55-eaca-48c7-aa38-236e04cf0dff-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25\" (UID: \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:56:39.788038 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.787848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfp6w\" (UniqueName: \"kubernetes.io/projected/e74d2c55-eaca-48c7-aa38-236e04cf0dff-kube-api-access-xfp6w\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25\" (UID: \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:56:39.791659 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.791637    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 28 18:56:39.800867 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.800836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e74d2c55-eaca-48c7-aa38-236e04cf0dff-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25\" (UID: \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:56:39.809977 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.809958    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 28 18:56:39.819905 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.819835    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 28 18:56:39.830084 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.830061    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfp6w\" (UniqueName: \"kubernetes.io/projected/e74d2c55-eaca-48c7-aa38-236e04cf0dff-kube-api-access-xfp6w\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25\" (UID: \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:56:39.888568 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.888535    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-hsm69\""
Jul 28 18:56:39.898836 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.898806    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-6fjp8\""
Jul 28 18:56:39.906995 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:39.906970    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:56:40.051491 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:40.051453    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"]
Jul 28 18:56:40.056114 ip-10-0-133-142 kubenswrapper[2559]: W0728 18:56:40.056080    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74d2c55_eaca_48c7_aa38_236e04cf0dff.slice/crio-b3f5ae6514553af2c246abdfb7fc72595b2e3f29df088a1b69a16a250ed3a2dc WatchSource:0}: Error finding container b3f5ae6514553af2c246abdfb7fc72595b2e3f29df088a1b69a16a250ed3a2dc: Status 404 returned error can't find the container with id b3f5ae6514553af2c246abdfb7fc72595b2e3f29df088a1b69a16a250ed3a2dc
Jul 28 18:56:40.983444 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:40.983406    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25" event={"ID":"e74d2c55-eaca-48c7-aa38-236e04cf0dff","Type":"ContainerStarted","Data":"a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f"}
Jul 28 18:56:40.983983 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:40.983458    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25" event={"ID":"e74d2c55-eaca-48c7-aa38-236e04cf0dff","Type":"ContainerStarted","Data":"b3f5ae6514553af2c246abdfb7fc72595b2e3f29df088a1b69a16a250ed3a2dc"}
Jul 28 18:56:40.983983 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:40.983480    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:56:42.975646 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:42.975610    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:49.967157 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:49.967127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:50.052435 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:56:50.052400    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:57:11.990613 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:57:11.990521    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:57:12.020363 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:57:12.020286    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25" podStartSLOduration=33.020264936 podStartE2EDuration="33.020264936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:56:41.043911196 +0000 UTC m=+5641.698406170" watchObservedRunningTime="2026-07-28 18:57:12.020264936 +0000 UTC m=+5672.674759912"
Jul 28 18:57:26.437325 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:57:26.437282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:57:29.735527 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:57:29.735484    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:57:30.956108 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:57:30.956076    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:57:59.370089 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:57:59.370057    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:57:59.382994 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:57:59.382953    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"]
Jul 28 18:57:59.440937 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:57:59.440895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:58:01.990464 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:01.990426    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25" podUID="e74d2c55-eaca-48c7-aa38-236e04cf0dff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:58:11.990330 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:11.990288    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25" podUID="e74d2c55-eaca-48c7-aa38-236e04cf0dff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:58:14.402574 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:14.402530    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25" podUID="e74d2c55-eaca-48c7-aa38-236e04cf0dff" containerName="main" containerID="cri-o://a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f" gracePeriod=45
Jul 28 18:58:14.647139 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:14.647114    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:58:14.714083 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:14.713994    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfp6w\" (UniqueName: \"kubernetes.io/projected/e74d2c55-eaca-48c7-aa38-236e04cf0dff-kube-api-access-xfp6w\") pod \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\" (UID: \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\") "
Jul 28 18:58:14.714083 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:14.714079    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e74d2c55-eaca-48c7-aa38-236e04cf0dff-tls-certs\") pod \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\" (UID: \"e74d2c55-eaca-48c7-aa38-236e04cf0dff\") "
Jul 28 18:58:14.716327 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:14.716297    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74d2c55-eaca-48c7-aa38-236e04cf0dff-kube-api-access-xfp6w" (OuterVolumeSpecName: "kube-api-access-xfp6w") pod "e74d2c55-eaca-48c7-aa38-236e04cf0dff" (UID: "e74d2c55-eaca-48c7-aa38-236e04cf0dff"). InnerVolumeSpecName "kube-api-access-xfp6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:58:14.716450 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:14.716366    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74d2c55-eaca-48c7-aa38-236e04cf0dff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e74d2c55-eaca-48c7-aa38-236e04cf0dff" (UID: "e74d2c55-eaca-48c7-aa38-236e04cf0dff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:58:14.814825 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:14.814784    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfp6w\" (UniqueName: \"kubernetes.io/projected/e74d2c55-eaca-48c7-aa38-236e04cf0dff-kube-api-access-xfp6w\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:58:14.814825 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:14.814819    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e74d2c55-eaca-48c7-aa38-236e04cf0dff-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 18:58:15.416081 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.416046    2559 generic.go:356] "Generic (PLEG): container finished" podID="e74d2c55-eaca-48c7-aa38-236e04cf0dff" containerID="a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f" exitCode=0
Jul 28 18:58:15.416519 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.416098    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25" event={"ID":"e74d2c55-eaca-48c7-aa38-236e04cf0dff","Type":"ContainerDied","Data":"a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f"}
Jul 28 18:58:15.416519 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.416138    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25" event={"ID":"e74d2c55-eaca-48c7-aa38-236e04cf0dff","Type":"ContainerDied","Data":"b3f5ae6514553af2c246abdfb7fc72595b2e3f29df088a1b69a16a250ed3a2dc"}
Jul 28 18:58:15.416519 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.416158    2559 scope.go:117] "RemoveContainer" containerID="a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f"
Jul 28 18:58:15.416519 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.416101    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"
Jul 28 18:58:15.425732 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.425709    2559 scope.go:117] "RemoveContainer" containerID="a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f"
Jul 28 18:58:15.426015 ip-10-0-133-142 kubenswrapper[2559]: E0728 18:58:15.425995    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f\": container with ID starting with a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f not found: ID does not exist" containerID="a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f"
Jul 28 18:58:15.426068 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.426026    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f"} err="failed to get container status \"a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f\": rpc error: code = NotFound desc = could not find container \"a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f\": container with ID starting with a0a4bc81a566a47d45023e919015c0a951b84151268d87fafb05ff5c86ccda9f not found: ID does not exist"
Jul 28 18:58:15.437639 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.437616    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"]
Jul 28 18:58:15.440969 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.440948    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-xxn25"]
Jul 28 18:58:15.834038 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:15.833949    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e74d2c55-eaca-48c7-aa38-236e04cf0dff" path="/var/lib/kubelet/pods/e74d2c55-eaca-48c7-aa38-236e04cf0dff/volumes"
Jul 28 18:58:20.409807 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:20.409768    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:58:21.468995 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:21.468960    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:58:26.633826 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:26.633787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:58:32.371998 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:32.371962    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:58:32.426846 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:58:32.426817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:59:40.488169 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:59:40.488110    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:59:48.093653 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:59:48.093618    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:59:54.003176 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:59:54.003129    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 18:59:54.103191 ip-10-0-133-142 kubenswrapper[2559]: I0728 18:59:54.103157    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:02:37.045139 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:02:37.045096    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:05:15.770988 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:15.770952    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"]
Jul 28 19:05:15.809779 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:15.809742    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:05:17.437637 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:17.437569    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:05:20.467653 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:20.467613    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5" podUID="0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 19:05:30.468408 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:30.468356    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5" podUID="0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 19:05:30.792115 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:30.791995    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5" podUID="0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" containerName="main" containerID="cri-o://b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d" gracePeriod=45
Jul 28 19:05:31.051274 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.051211    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 19:05:31.110641 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.110610    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8sbw\" (UniqueName: \"kubernetes.io/projected/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-kube-api-access-m8sbw\") pod \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\" (UID: \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\") "
Jul 28 19:05:31.110821 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.110676    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-tls-certs\") pod \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\" (UID: \"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47\") "
Jul 28 19:05:31.113056 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.113024    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-kube-api-access-m8sbw" (OuterVolumeSpecName: "kube-api-access-m8sbw") pod "0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" (UID: "0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47"). InnerVolumeSpecName "kube-api-access-m8sbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:05:31.113056 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.113037    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" (UID: "0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:05:31.211530 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.211491    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8sbw\" (UniqueName: \"kubernetes.io/projected/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-kube-api-access-m8sbw\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 19:05:31.211530 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.211523    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47-tls-certs\") on node \"ip-10-0-133-142.ec2.internal\" DevicePath \"\""
Jul 28 19:05:31.300342 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.300307    2559 generic.go:356] "Generic (PLEG): container finished" podID="0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" containerID="b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d" exitCode=0
Jul 28 19:05:31.300511 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.300362    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"
Jul 28 19:05:31.300511 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.300390    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5" event={"ID":"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47","Type":"ContainerDied","Data":"b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d"}
Jul 28 19:05:31.300511 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.300434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5" event={"ID":"0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47","Type":"ContainerDied","Data":"d37fc85713eaafbb28608557d559f5b402226abd58eeadc03922d3506baa921f"}
Jul 28 19:05:31.300511 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.300455    2559 scope.go:117] "RemoveContainer" containerID="b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d"
Jul 28 19:05:31.310213 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.310191    2559 scope.go:117] "RemoveContainer" containerID="b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d"
Jul 28 19:05:31.310496 ip-10-0-133-142 kubenswrapper[2559]: E0728 19:05:31.310475    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d\": container with ID starting with b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d not found: ID does not exist" containerID="b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d"
Jul 28 19:05:31.310547 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.310507    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d"} err="failed to get container status \"b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d\": rpc error: code = NotFound desc = could not find container \"b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d\": container with ID starting with b79163f11910a613b4606b1d0a206007233103e0e3378b31614c3826fe9aed4d not found: ID does not exist"
Jul 28 19:05:31.322259 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.322232    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"]
Jul 28 19:05:31.329130 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.329105    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5"]
Jul 28 19:05:31.832447 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:31.832417    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" path="/var/lib/kubelet/pods/0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47/volumes"
Jul 28 19:05:34.771334 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:34.771299    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:05:34.830163 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:34.830127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:05:58.129929 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:58.129893    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-82lr6_adb2854c-ed88-4cd3-bb49-e4436c4f02ff/discovery/0.log"
Jul 28 19:05:58.145299 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:58.145269    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/0.log"
Jul 28 19:05:58.188286 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:58.188255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/1.log"
Jul 28 19:05:59.257190 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:59.257154    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-82lr6_adb2854c-ed88-4cd3-bb49-e4436c4f02ff/discovery/0.log"
Jul 28 19:05:59.276203 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:59.276152    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/0.log"
Jul 28 19:05:59.319746 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:05:59.319707    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/1.log"
Jul 28 19:06:00.363523 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:00.363489    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-82lr6_adb2854c-ed88-4cd3-bb49-e4436c4f02ff/discovery/0.log"
Jul 28 19:06:00.380361 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:00.380327    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/0.log"
Jul 28 19:06:00.422294 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:00.422255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/1.log"
Jul 28 19:06:01.488990 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:01.488957    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-82lr6_adb2854c-ed88-4cd3-bb49-e4436c4f02ff/discovery/0.log"
Jul 28 19:06:01.537528 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:01.537499    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/0.log"
Jul 28 19:06:01.579557 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:01.579523    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/1.log"
Jul 28 19:06:02.566357 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:02.566325    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:02.795833 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:02.795807    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:03.921488 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:03.921454    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:04.254330 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:04.254239    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:05.274485 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:05.274450    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:05.476920 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:05.476888    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:06.460331 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:06.460302    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:06.665513 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:06.665477    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:07.648323 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:07.648288    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:07.857853 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:07.857815    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:08.833910 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:08.833858    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:09.038368 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:09.038331    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:10.044970 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:10.044930    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:10.248590 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:10.248553    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:11.218891 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:11.218836    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:11.425630 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:11.425599    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:12.420624 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:12.420581    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:12.621375 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:12.621328    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:13.633633 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:13.633603    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:13.845698 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:13.845620    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:14.819830 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:14.819801    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:15.056888 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:15.056838    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:16.026033 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:16.025994    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:06:16.081154 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:16.081109    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:16.125400 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:16.125349    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:06:16.275610 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:16.275579    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:17.175902 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:17.175848    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:17.390411 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:17.390380    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:18.387990 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:18.387956    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:18.591000 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:18.590968    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:19.603965 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:19.603933    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:19.801725 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:19.801687    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:20.786933 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:20.786902    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:21.002869 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:21.002830    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:22.110054 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:22.110021    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:22.321895 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:22.321847    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:23.303663 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:23.303627    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:23.503925 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:23.503869    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:24.489368 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:24.489335    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:24.685335 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:24.685303    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:25.660371 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:25.660340    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:25.882746 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:25.882715    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:26.872646 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:26.872611    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:27.077317 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:27.077294    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:28.066582 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:28.066550    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:28.278672 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:28.278640    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:29.306704 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:29.306668    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:29.516045 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:29.516015    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:30.522565 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:30.522531    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:30.748036 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:30.748008    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:31.829798 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:31.829766    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:32.040283 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:32.040247    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:34.062046 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:34.062014    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:06:34.230636 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:34.230593    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:06:34.291333 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:34.291296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:06:37.188127 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:37.188092    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rff9b_1c6f3233-64d4-424d-9f24-b583d6beacf7/global-pull-secret-syncer/0.log"
Jul 28 19:06:37.337734 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:37.337681    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log"
Jul 28 19:06:37.568791 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:37.568700    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-142.ec2.internal_f12b46d1950955bc6884ff5ac3c8c404/haproxy/0.log"
Jul 28 19:06:40.132955 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:40.132918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:06:40.233035 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:40.232991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2ffrd_e4153a55-9fee-476f-a80a-155d275f14e1/konnectivity-agent/0.log" limit=1024
Jul 28 19:06:41.908545 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:41.908479    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-r8p5c_a34b85d0-c9c1-424c-9479-b130b8aa5d71/manager/0.log"
Jul 28 19:06:42.279443 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:42.279357    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-dzzbw_60c6f8b3-8313-4b72-a03d-0399f41c5aa8/manager/0.log"
Jul 28 19:06:44.310507 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:44.310472    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4nmcq_c969b397-8973-4a66-be48-9911f7e203e7/node-exporter/0.log"
Jul 28 19:06:44.340808 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:44.340777    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4nmcq_c969b397-8973-4a66-be48-9911f7e203e7/kube-rbac-proxy/0.log"
Jul 28 19:06:44.362578 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:44.362559    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4nmcq_c969b397-8973-4a66-be48-9911f7e203e7/init-textfile/0.log"
Jul 28 19:06:48.223949 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.223907    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"]
Jul 28 19:06:48.224456 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.224353    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" containerName="main"
Jul 28 19:06:48.224456 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.224371    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" containerName="main"
Jul 28 19:06:48.224456 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.224383    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74d2c55-eaca-48c7-aa38-236e04cf0dff" containerName="main"
Jul 28 19:06:48.224456 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.224391    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74d2c55-eaca-48c7-aa38-236e04cf0dff" containerName="main"
Jul 28 19:06:48.224719 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.224480    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74d2c55-eaca-48c7-aa38-236e04cf0dff" containerName="main"
Jul 28 19:06:48.224719 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.224492    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c19ee5c-5ca7-48bd-9a1f-951af3fbbf47" containerName="main"
Jul 28 19:06:48.227594 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.227572    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.230112 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.230089    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rd9v9\"/\"default-dockercfg-k9wgm\""
Jul 28 19:06:48.230244 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.230088    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd9v9\"/\"openshift-service-ca.crt\""
Jul 28 19:06:48.230244 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.230132    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd9v9\"/\"kube-root-ca.crt\""
Jul 28 19:06:48.238181 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.238157    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"]
Jul 28 19:06:48.332492 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.332450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxrlf\" (UniqueName: \"kubernetes.io/projected/439a23ca-ff6c-4373-832b-7b4ec274ba36-kube-api-access-lxrlf\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.332711 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.332518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-proc\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.332711 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.332554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-lib-modules\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.332711 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.332581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-podres\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.332711 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.332627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-sys\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.433186 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.433143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxrlf\" (UniqueName: \"kubernetes.io/projected/439a23ca-ff6c-4373-832b-7b4ec274ba36-kube-api-access-lxrlf\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.433399 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.433202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-proc\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.433399 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.433237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-lib-modules\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.433399 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.433264    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-podres\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.433399 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.433304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-proc\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.433399 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.433324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-sys\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.437893 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.433764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-sys\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.437893 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.433428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-lib-modules\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.437893 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.433429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/439a23ca-ff6c-4373-832b-7b4ec274ba36-podres\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.442781 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.442755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxrlf\" (UniqueName: \"kubernetes.io/projected/439a23ca-ff6c-4373-832b-7b4ec274ba36-kube-api-access-lxrlf\") pod \"perf-node-gather-daemonset-dfh6x\" (UID: \"439a23ca-ff6c-4373-832b-7b4ec274ba36\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.538408 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.538307    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:48.676431 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:48.676401    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"]
Jul 28 19:06:48.679118 ip-10-0-133-142 kubenswrapper[2559]: W0728 19:06:48.679089    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod439a23ca_ff6c_4373_832b_7b4ec274ba36.slice/crio-3b865fa918688031aee0b6e5b2dcb4edaf9c27a480533d223729b173088961fb WatchSource:0}: Error finding container 3b865fa918688031aee0b6e5b2dcb4edaf9c27a480533d223729b173088961fb: Status 404 returned error can't find the container with id 3b865fa918688031aee0b6e5b2dcb4edaf9c27a480533d223729b173088961fb
Jul 28 19:06:49.620833 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:49.620798    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-496j7_9410f0ea-7664-4c08-986d-7950cfcabfc7/dns/0.log"
Jul 28 19:06:49.639293 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:49.639253    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x" event={"ID":"439a23ca-ff6c-4373-832b-7b4ec274ba36","Type":"ContainerStarted","Data":"cf2694409c63d92ba99c073af2e5707a3b01918544e0749aca34687ce083a2c7"}
Jul 28 19:06:49.639483 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:49.639317    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x" event={"ID":"439a23ca-ff6c-4373-832b-7b4ec274ba36","Type":"ContainerStarted","Data":"3b865fa918688031aee0b6e5b2dcb4edaf9c27a480533d223729b173088961fb"}
Jul 28 19:06:49.639483 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:49.639344    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:49.663057 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:49.663003    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x" podStartSLOduration=1.662986066 podStartE2EDuration="1.662986066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:06:49.661955296 +0000 UTC m=+6250.316450262" watchObservedRunningTime="2026-07-28 19:06:49.662986066 +0000 UTC m=+6250.317481060"
Jul 28 19:06:49.664383 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:49.664357    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-496j7_9410f0ea-7664-4c08-986d-7950cfcabfc7/kube-rbac-proxy/0.log"
Jul 28 19:06:49.793312 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:49.793279    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mtzfw_98966b68-fcf4-411b-9db9-20fd9012cd26/dns-node-resolver/0.log"
Jul 28 19:06:50.428071 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:50.428037    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-986jm_4382b057-9b5d-43fb-8d3e-a64b33732a92/node-ca/0.log"
Jul 28 19:06:51.867445 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:51.867414    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-82lr6_adb2854c-ed88-4cd3-bb49-e4436c4f02ff/discovery/0.log"
Jul 28 19:06:51.891592 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:51.891548    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/0.log"
Jul 28 19:06:51.934239 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:51.934208    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/1.log"
Jul 28 19:06:52.483476 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:52.483445    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6wjcw_5edec3e6-cd2b-40b0-a939-e91932b835c8/serve-healthcheck-canary/0.log"
Jul 28 19:06:53.108308 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:53.108275    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nvtkq_9979e836-b374-4a29-b553-486da46ce962/kube-rbac-proxy/0.log"
Jul 28 19:06:53.131067 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:53.131043    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nvtkq_9979e836-b374-4a29-b553-486da46ce962/exporter/0.log"
Jul 28 19:06:53.156522 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:53.156493    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nvtkq_9979e836-b374-4a29-b553-486da46ce962/extractor/0.log"
Jul 28 19:06:55.654169 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:55.654136    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-dfh6x"
Jul 28 19:06:57.168674 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:57.168646    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-hwv4w_6fe48c5b-b209-4020-bce4-dcd8406038a9/server/0.log"
Jul 28 19:06:58.420615 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:58.420580    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-t2bz8_6a305e10-351c-48eb-95af-1dd929296f5a/manager/0.log"
Jul 28 19:06:58.460741 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:58.460706    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-82lr6_adb2854c-ed88-4cd3-bb49-e4436c4f02ff/discovery/0.log"
Jul 28 19:06:58.480336 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:58.480301    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/0.log"
Jul 28 19:06:58.522323 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:58.522276    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/1.log"
Jul 28 19:06:58.528657 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:06:58.528633    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-db8st_f4c353be-e8d5-4191-9193-21c36a64f7ae/seaweedfs/0.log"
Jul 28 19:07:00.020274 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:00.020239    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-82lr6_adb2854c-ed88-4cd3-bb49-e4436c4f02ff/discovery/0.log"
Jul 28 19:07:00.047909 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:00.047822    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/0.log"
Jul 28 19:07:00.089625 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:00.089584    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-pfnkg_67fbff38-b30c-4641-95cc-f266c7219170/istio-proxy/1.log"
Jul 28 19:07:05.196108 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.196026    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxvgt_68af73e0-884e-457e-9e5b-17a3ded1a5c3/kube-multus-additional-cni-plugins/0.log"
Jul 28 19:07:05.257525 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.257494    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxvgt_68af73e0-884e-457e-9e5b-17a3ded1a5c3/egress-router-binary-copy/0.log"
Jul 28 19:07:05.316615 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.316545    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxvgt_68af73e0-884e-457e-9e5b-17a3ded1a5c3/cni-plugins/0.log"
Jul 28 19:07:05.344769 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.344741    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxvgt_68af73e0-884e-457e-9e5b-17a3ded1a5c3/bond-cni-plugin/0.log"
Jul 28 19:07:05.367199 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.367170    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxvgt_68af73e0-884e-457e-9e5b-17a3ded1a5c3/routeoverride-cni/0.log"
Jul 28 19:07:05.396370 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.396343    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxvgt_68af73e0-884e-457e-9e5b-17a3ded1a5c3/whereabouts-cni-bincopy/0.log"
Jul 28 19:07:05.420789 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.420759    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dxvgt_68af73e0-884e-457e-9e5b-17a3ded1a5c3/whereabouts-cni/0.log"
Jul 28 19:07:05.880097 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.880045    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zcfck_020a8f3e-cfe7-4a8e-b390-3b8c67ca4d8b/kube-multus/0.log"
Jul 28 19:07:05.904814 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.904784    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-89z9z_39014b65-63d6-416a-90bd-9baf56efe171/network-metrics-daemon/0.log"
Jul 28 19:07:05.925757 ip-10-0-133-142 kubenswrapper[2559]: I0728 19:07:05.925728    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-89z9z_39014b65-63d6-416a-90bd-9baf56efe171/kube-rbac-proxy/0.log"